Methods, systems and devices for providing portions of recorded game content in response to an audio trigger

ABSTRACT

A system that incorporates the subject disclosure may include, for example, a gaming system that cooperates with a graphical user interface to enable user modification and enhancement of one or more audio streams associated with the gaming system. In embodiments, the audio streams may include a game audio stream, a chat audio stream of conversation among players of a video game, and a microphone audio stream of a player of the video game. One of the audio streams may trigger an action in the gaming system such as recording a portion of game content for a timer period. Additional embodiments are disclosed.

CROSS-REFERENCE TO RELATED APPLICATION(S)

The present application claims the benefit of priority to U.S.Provisional Application No. 63/240,004 filed Sep. 2, 2021. All sectionsof the aforementioned application are incorporated herein by referencein their entirety.

FIELD OF THE DISCLOSURE

The present disclosure relates generally to methods, systems, anddevices of providing portions of recorded game content in response to anaudio trigger.

BACKGROUND

It is common today for gamers to utilize more than one gaming accessory.This is especially true of gamers who play on-line games or competitivegames in a team or individual configuration. Gamers can have at theirdisposal accessories such as a keyboard, a general purpose gaming pad, amouse, a gaming console controller, a headset to communicate with otherplayers, a joystick, a computer console, or other common gamingaccessories.

A gamer can frequently use a combination of these accessories in asingle game (e.g., headset, a keyboard, and mouse). Efficient managementand utilization of these accessories can frequently impact a gamer'sability to compete.

Accessory management can have utility in other disciplines which may notrelate to gaming applications. Efficient use of accessories in theseother disciplines can be important to other users.

In addition, a player can play a video game with other players, asteammates or against opponents, all of which are remotely connected overa communication network. Players may communicate with teammates viaspoken audio and players may hear game sounds generated by the videogame. Further, the video game can monitor aspects of player interactionwith the video game such that player performance can be reviewed toimprove future performance.

BRIEF DESCRIPTION OF THE DRAWINGS

Reference will now be made to the accompanying drawings, which are notnecessarily drawn to scale, and wherein:

FIG. 1 depicts an illustrative embodiment of a Graphical User Interface(GUI) generated by an Accessory Management Software (AMS) applicationaccording to the present disclosure;

FIGS. 2-3 depict illustrative embodiments for communicatively coupling agaming controller to a computing device;

FIG. 4 depicts an illustrative embodiment of a communication device;

FIG. 5 depicts an illustrative embodiment of a first method utilized inthe subject disclosure;

FIG. 6 depicts an illustrative embodiment of a second method utilized inthe subject disclosure;

FIG. 7A depicts an illustrative embodiment of a third method utilized inthe subject disclosure;

FIG. 7B depicts an illustrative embodiment of a fourth method utilizedin the subject disclosure;

FIG. 7C depicts an illustrative embodiment of a fifth method utilized inthe subject disclosure;

FIG. 7D depicts an illustrative embodiment of a sixth method utilized inthe subject disclosure;

FIGS. 7E-7G depict illustrative embodiments of interfaces associatedwith a gameplay of a game;

FIG. 7H depicts an illustrative embodiment of a method for automatictriggering of a notification to one or more gameplayers based on adetected sound, in accordance with aspects of the subject disclosure;

FIG. 7I depicts an illustrative embodiment of an additional method forautomatic triggering of a notification to one or more gameplayers basedon a detected sound, in accordance with aspects of the subjectdisclosure;

FIGS. 7J and 7K schematically illustrate a system operating at least inpart according to the methods of FIGS. 5, 6, 7A-7C and 7H-7I;

FIG. 7L depicts an illustrative embodiment of a block diagram for anaudio processing system for a gaming system in in accordance withaspects of the subject disclosure;

FIGS. 7L-1 and FIG. 7L-2 depict an illustrative embodiment of operatingsystem controls of a user interface for selecting device preferences;

FIG. 7L-3 depict an illustrative embodiment of a process for listenerselection of a head-related transfer function (HRTF) profile for use bythe user;

FIGS. 7M-7T depict an illustrative embodiment of a graphical userinterface display of an audio processing system for a gaming system inaccordance with aspects of the subject disclosure;

FIG. 8 depicts an illustrative embodiment of a system operating at leastin part according to the methods of FIGS. 5-7 ;

FIG. 9 depicts an illustrative embodiment of a communication flowdiagram utilized by the system of FIG. 12 ; and

FIG. 10 depicts an illustrative diagrammatic representation of a machinein the form of a computer system within which a set of instructions,when executed, may cause the machine to perform any one or more of themethodologies disclosed herein.

FIG. 11 depicts an illustrative diagrammatic representation of a slidingwindow in accordance with aspects of this disclosure.

FIG. 12 depicts the storage media for storing a representation of asliding window in accordance with aspects of this disclosure.

FIG. 13 depicts system to identify a group of possible trigger clips inaccordance with aspects of this disclosure.

FIG. 14 depicts a system to distribute a group of trigger clips to agroup of players in accordance with aspects of this disclosure.

FIGS. 15A-D depict illustrative embodiments of methods utilized in thesubject disclosure.

FIGS. 15E and 15F depict illustrative embodiments of methods utilized inthe subject disclosure

DETAILED DESCRIPTION

The subject disclosure describes, among other things, illustrativeembodiments of, for example, a gaming system that cooperates with agraphical user interface to enable user modification and enhancement ofone or more audio streams associated with the gaming system. Inembodiments, the audio streams may include a game audio stream, a chataudio stream of conversation among players of a video game, and amicrophone audio stream of a player of the video game. A player mayadjust audio of the audio streams independently to improve clarity,reduce noise, assist in identifying in-game sounds such as footsteps andgunshots. The graphical user interface may be displayed on a displaydevice of the player and manually adjusted to preferred settings orplayer profile or configuration file may be retrieved for each audiostream. To further enhance sound and audio quality for a player, ahead-related transfer function may be chosen from among a set ofpredefined transfer functions. Audio may be provided in a surround soundarrangement to enhance the audio experience of the user still further.Individual sounds within the game audio may be analyzed and identified,such as by a neural network, and enhanced to give the game player animproved audio experience and a competitive advantage in playing thevideo game. Other embodiments are described in the subject disclosure.

One embodiment of the subject disclosure includes receiving a pluralityof audio streams including a game output audio stream of a gamingapplication, a gamer chat audio stream produced by a plurality of gamersparticipating in the gaming application, and a gamer microphone audiostream produced by one gamer of the plurality of gamers, processing thegame output audio stream according to first audio processing settings ofone gamer, forming a processed game output audio stream, and processingthe gamer chat audio stream according to second audio processingsettings of the one gamer, forming a processed gamer chat audio stream.Embodiments further include processing the gamer microphone audio streamaccording to third audio processing settings of the one gamer, forming aprocessed gamer microphone audio stream, providing the processed gameoutput audio stream and the processed gamer chat audio stream to audioequipment of the one gamer, and providing, the processed gamermicrophone audio stream to one or more gamers of the plurality ofgamers.

One embodiment of the subject disclosure includes receiving a live gameoutput audio stream of a gaming application currently being played bythe plurality of gamers, providing on a display device of one gamer ofthe plurality of gamers, a graphical user interface, the graphical userinterface including a frequency spectrum analyzer interface forselection of one or more live audio processing settings by the one gamerfor the live game output audio stream, receiving from the graphical userinterface, data defining the one or more live audio processing settings,and modifying the live game output audio stream of the gamingapplication according to the one or more live audio processing settings.

One embodiment of the subject disclosure includes receiving a live gameoutput audio stream of a gaming application currently being played bythe plurality of gamers, retrieving a stored configuration file, thestored configuration file including data defining stored live audioprocessing settings, and modifying the live game output audio stream ofthe gaming application according to the stored live audio processingsettings.

One embodiment of the subject disclosure includes a processing systemincluding a processor and a memory that stores executable instructionsthat, when executed by the processing system, facilitate performance ofoperations, the operations comprising a gaming application producinggaming output information including a game output audio stream and anaudio processing module in data communication with the gamingapplication, the audio processing module operative to receive the gameoutput audio stream from the gaming application and to produce aplurality of game audio streams, the audio processing module furtheroperative to apply respective audio customizations to each respectivegame audio stream of the plurality of game audio streams, according togamer input defining the respective audio customizations for eachrespective game audio stream.

One embodiment of the subject disclosure includes a device including agaming application producing gaming output information including a gameoutput audio stream and a head-related transfer function (HRTF) modulein data communication with the gaming application, the HRTF moduleoperative to receive the game output audio stream from the gamingapplication and to apply a selected head-related transfer function tothe game output audio stream to produce a surround sound audio stream,the selected head-related transfer function selected from a plurality ofstandardized head-related transfer functions.

One embodiment of the subject disclosure includes receiving a gamingsystem audio stream during a video game, detecting a current trigger inthe gaming system audio stream during the video game, recording atrigger clip of game content, wherein the trigger clip comprises aportion of game content that includes the current trigger, and providingthe clip to a group of gaming devices over a communication network,wherein each gaming device of the group of gaming devices is associatedwith one player of the video game.

One embodiment of the subject disclosure includes receiving a gamingsystem audio stream during a video game the gaming system audio streamcomprising a plurality of audio streams including a chat audio stream, agame audio stream and a microphone audio stream, detecting, by theprocessing system, a predetermined event in the gaming system audiostream during the video game, modifying an audio spectrum of the gamingsystem audio stream to emphasize the predetermined event in the gamingsystem audio stream, producing an equalized audio stream, and providingthe equalized audio stream to an audio device of a player of the videogame.

FIG. 1 depicts an illustrative embodiment of a Graphical User Interface(GUI) generated by an Accessory Management Software (AMS) applicationaccording to the present disclosure. The AMS application can be executedby a computing device such as a desktop computer, a laptop computer, atablet, a server, a mainframe computer, a gaming console, a gamingaccessory, or any combination or portions thereof. The AMS applicationcan also be executed by portable computing devices such as a cellularphone, a personal digital assistant, or a media player. The AMSapplication can be executed by any device with suitable computing andcommunication resources.

FIG. 2 illustrates a number of embodiments for utilizing a gamingcontroller 115 with a computing device 206 in the form of a gamingconsole. In the illustration of FIG. 2 , the gaming controller 115 canbe communicatively coupled to the gaming console 206 with a tetheredcable interface 202 such as a USB or proprietary cable, or a wirelessinterface 204 such as Wi-Fi, Bluetooth, ZigBee, or a proprietarywireless communications protocol. The cable interface 202 provides ameans for communication that may be less susceptible to electromagneticinterference. It will be appreciated that the gaming controller 115 mayfurther include a headset 114 (with or without a microphone not shown)utilized by a gamer to communicate with teammates and/or to listen togame sounds in high fidelity. In the illustration of FIG. 2 , the AMSapplication can in whole or in part be executed by the gaming controller115, the gaming console 206, or a combination thereof.

FIG. 3 illustrates a number of other embodiments for utilizing a gamingcontroller 115 with a computing device 206. In this embodiment, thegaming controller 115 comprises a mouse and the computing device 206comprises a computer. The gaming controller 115 can be tethered to thecomputing device 206 by a cable interface 202 (e.g., USB cable orproprietary cable) or a wireless interface 204. The cable interface 202provides a means for communication that may be less susceptible toelectromagnetic interference. It will be appreciated that the gamingcontroller 115 may further include a headset (with or without amicrophone not shown) utilized by a gamer to communicate with teammatesand/or to listen to game sounds in high fidelity. In the illustration ofFIG. 3 , the AMS application can in whole or in part be executed by thegaming controller 115, the gaming console 206, or a combination thereof.

For illustration purposes, the terms gaming console 206 and computer 206will be used hence forth interchangeably with the term computing device206 with an understanding that a computing device 206 may represent anumber of other devices such as a server, a tablet, a smart phone, andso on. Accordingly, a computing device 206 can represent any device withsuitable computing resources to perform the methods described in thesubject disclosure.

FIG. 4 depicts an illustrative embodiment of a communication device 400.Communication device 400 can serve in whole or in part as anillustrative embodiment of devices described in the subject disclosure.The communication device 400 can comprise a wireline and/or wirelesstransceiver 402 (herein transceiver 402), a user interface (UI) 404, apower supply 414, a proximity sensor 416, a motion sensor 418, anorientation sensor 420, and a controller 406 for managing operationsthereof. The transceiver 402 can support short-range or long-rangewireless access technologies such as Bluetooth, Wi-Fi, Digital EnhancedCordless Telecommunications (DECT), or cellular communicationtechnologies, just to mention a few. Cellular technologies can include,for example, CDMA-1X, UMTS/HSDPA, GSM/GPRS, TDMA/EDGE, EV/DO, WiMAX,software defined radio (SDR), Long Term Evolution (LTE), as well asother next generation wireless communication technologies as they arise.The transceiver 402 can also be adapted to support circuit-switchedwireline access technologies (such as PSTN), packet-switched wirelineaccess technologies (such as TCP/IP, VoIP, etc.), and combinationsthereof.

The UI 404 can include a depressible or touch-sensitive keypad 408coupled to a navigation mechanism such as a roller ball, a joystick, amouse, or a navigation disk for manipulating operations of thecommunication device 400. The keypad 408 can be an integral part of ahousing assembly of the communication device 400 or an independentdevice operably coupled thereto by a tethered wireline interface (suchas a USB cable) or a wireless interface supporting for exampleBluetooth. The keypad 408 can represent a numeric keypad, and/or aQWERTY keypad with alphanumeric keys. The UI 404 can further include adisplay 410 such as monochrome or color LCD (Liquid Crystal Display),OLED (Organic Light Emitting Diode) or other suitable display technologyfor conveying images to an end user of the communication device 400.

In an embodiment where the display 410 utilizes touch-sensitivetechnology, a portion or all of the keypad 408 can be presented by wayof the display 410 with navigation features. As a touch screen display,the communication device 400 can be adapted to present a user interfacewith graphical user interface (GUI) elements that can be selected by auser with a touch of a finger. The touch screen display 410 can beequipped with capacitive, resistive or other forms of sensing technologyto detect how much surface area of a user's finger has been placed on aportion of the touch screen display. This sensing information can beused to control the manipulation of the GUI elements.

The UI 404 can also include an audio system 412 that utilizes commonaudio technology for conveying low volume audio (such as audio heardonly in the proximity of a human ear) and high-volume audio (such asspeakerphone for hands free operation, stereo or surround sound system).The audio system 412 can further include a microphone for receivingaudible signals of an end user. The audio system 412 can also be usedfor voice recognition applications. The UI 404 can further include animage sensor 413 such as a charged coupled device (CCD) camera forcapturing still or moving images and performing image recognitiontherefrom.

The power supply 414 can utilize common power management technologiessuch as replaceable or rechargeable batteries, supply regulationtechnologies, and charging system technologies for supplying energy tothe components of the communication device 400 to facilitate long-rangeor short-range portable applications. Alternatively, the charging systemcan utilize external power sources such as DC power supplied over aphysical interface such as a USB port or by way of a power cord attachedto a transformer that converts AC to DC power.

The proximity sensor 416 can utilize proximity sensing technology suchas an electromagnetic sensor, a capacitive sensor, an inductive sensor,an image sensor or combinations thereof. The motion sensor 418 canutilize motion sensing technology such as an accelerometer, a gyroscope,or other suitable motion sensing technology to detect movement of thecommunication device 400 in three-dimensional space. The orientationsensor 420 can utilize orientation sensing technology such as amagnetometer to detect the orientation of the communication device 400(North, South, West, East, combined orientations thereof in degrees,minutes, or other suitable orientation metrics).

The communication device 400 can use the transceiver 402 to alsodetermine a proximity to a cellular, Wi-Fi, Bluetooth, or other wirelessaccess points by common sensing techniques such as utilizing a receivedsignal strength indicator (RSSI) and/or a signal time of arrival (TOA)or time of flight (TOF). The controller 406 can utilize computingtechnologies such as a microprocessor, a digital signal processor (DSP),and/or a video processor with associated storage memory such as Flash,ROM, RAM, SRAM, DRAM or other storage technologies.

The communication device 400 as described herein can operate with moreor less components described in FIG. 4 to accommodate the implementationof devices described by the subject disclosure. These variantembodiments are contemplated by the subject disclosure.

FIGS. 5-7A depict methods 500-700 describing illustrative embodiments ofthe AMS application. Method 500 can begin with step 502 in which the AMSapplication is invoked in a computing device. The computing device canbe a remote server (not shown), the gaming console 206 or computer 206of FIGS. 2-3 , or any other computing device with suitable computingresources. The invocation step can result from a user selection of theAMS application from a menu or iconic symbol presented by the computingdevice 206, or when a user communicatively couples a gaming controller115 or other form of accessory device with the computing device 206. Instep 504, the AMS application can detect by way of software drivers inan operating system (OS) of the computing device 206 a plurality ofoperationally distinct accessories communicatively coupled to thecomputing device 206. The accessories can be coupled to the computingdevice 206 by a tethered interface (e.g., USB cable), a wirelessinterface (e.g., Bluetooth or Wireless Fidelity—Wi-Fi), or combinationsthereof.

In the present context, an accessory can represent any type of devicewhich can be communicatively coupled to the computing device 206 (orwhich can be an integral part of the computing device) and which cancontrol aspects of the OS and/or a software application operating fromthe computing device 206. An accessory can represent for example akeyboard, a touch screen display, a gaming pad, a gaming controller, amouse, a joystick, a microphone, or a headset with a microphone—just tomention a few.

In step 506, the AMS application presents a GUI 101 such as depicted inFIG. 1 depicting operationally distinct accessories such as a keyboard108, and a gaming controller 115. The GUI 101 presents the accessories108-116 in a scrollable section 117. One or more accessories can beselected by a user with a mouse pointer. In this illustration, thekeyboard 108 and the gaming controller 115 were selected forcustomization. Upon selecting the keyboard 108 and the gaming controller115 from the scrollable window of section 117, the AMS applicationpresents the keyboard 108 and the gaming controller 115 in split windows118, 120, respectively, to assist the user during the customizationprocess.

In step 508, the AMS application can be programmed to detect auser-selection of a particular software application such as a videogame. This step can be the result of the user entering in a Quick Searchfield 160 the name of a gaming application (e.g., World of Warcraft™ orWoW). Upon identifying a gaming application, the AMS application canretrieve in step 510 from a remote or local database gaming applicationaction which can be presented in a scrollable section 139 of the GUIrepresented as “Actions” 130. The actions can be tactical actions 132,communication actions 134, menu actions 136, and movement actions 138which can be used to invoke and manage features of the gamingapplication.

The actions presented descriptively in section 130 of the GUI canrepresent a sequence of accessory input functions which a user canstimulate by button depressions, navigation or speech. For example,depressing the left button on the mouse 110 can represent the tacticalaction “Reload”, while the simultaneous keyboard depressions “Ctrl A”can represent the tactical action “Melee Attack”. For ease of use, the“Actions” 130 section of the GUI is presented descriptively rather thanby a description of the input function(s) of a particular accessory.

Any one of the Actions 130 can be associated with one or more inputfunctions of the accessories being customized in windows 118 and 120 byway of a drag and drop action or other customization options. Forinstance, a user can select a “Melee Attack” by placing a mouse pointer133 over an iconic symbol associated with this action. Upon doing so,the symbol can be highlighted to indicate to the user that the icon isselectable. At this point, the user can select the icon by holding theleft mouse button and drag the symbol to any of the input functions(e.g., buttons) of the keyboard 108 or selectable options of the gamingcontroller 115 to make an association with an input function of one ofthese accessories. Actions of one accessory can also be associated withanother accessory that is of a different category. For example, keydepressions “Ctrl A” of the keyboard 108 can be associated with one ofthe buttons of the gaming controller 115 (e.g., the left button 119).

In one embodiment, a Melee Attack action can be associated by draggingthis action to either the left button 119 or right button 121 of thegaming controller 115. Thus, when the selected button is depressed, thestimulus signal that is generated by the selected button of the gamingcontroller 115 can be substituted by the AMS application with the MeleeAttack action. In another embodiment, the AMS application can beconfigured so that the Melee Action can be associated with a combinationof key button presses (e.g., simultaneous depression of the left andright buttons 119, 121, or a sequence of button depressions: two rapidleft button depressions followed by a right button depression).

In yet another embodiment, the AMS application can be configured so thatthe Melee Action can be associated with movement of the gamingcontroller 115 such as, for example, rapid movement or shaking of thegaming controller 115. In a further embodiment, the AMS application canbe adapted to make associations with two dimensional orthree-dimensional movements of the gaming controller 115 according to agaming venue state. For example, suppose the player's avatar enters afighter jet. In this gaming venue state, moving the left navigation knobforward can be associated by the AMS application with controlling thethrottle of the jet engines. Rapidly moving the gaming controller 115downward can represent release of munitions such as a bomb.

In a gaming venue state where the gamer's avatar has entered a building,lifting of the gaming controller 115 above a first displacementthreshold can be associated with a rapid movement of the avatar up onefloor. A second displacement threshold can be associated with a rapidmovement of the avatar down one floor—the opposite of the firstdisplacement threshold. Alternatively, the second displacement thresholdcould be associated with a different action such as jumping betweenbuildings when the avatar is on the roof of a building.

The AMS application can monitor gaming venue states by analyzingcaptured images produced by the gaming application (e.g., one or morestill images of a tank, or a video of an avatar entering a tank), and/orby receiving messages from the gaming application by way of anapplication programming interface (API) thereby enabling the AMSapplication to identify the occurrence of a particular gaming venuestate.

At step 512 the AMS application can also respond to a user selection ofa profile. A profile can be a device profile or master profile invokedby selecting GUI button 156 or 158, each of which can identify theassociation of gaming actions with input functions of one or moreaccessories. If a profile selection is detected in step 512, the AMSapplication can retrieve in step 514 macro(s) and/or prior associationsdefined by the profile. The actions and/or macros defined in the profilecan also be presented in step 516 by the AMS application in the actionscolumn 130 of the GUI 101 to modify existing profile associations orcreate new associations.

In step 518, the AMS application can also respond to a user selection tocreate a macro. A macro in the present context can mean any actionablecommand which can be recorded by the AMS application. An actionablecommand can represent a sequence of stimuli generated by manipulatinginput functions of an accessory, a combination of actions in the Actionsection 130, an identification of a software application to be initiatedby the OS of the computing device 206, or any other recordable stimulusto initiate, control or manipulate software applications. For instance,a macro can represent a user entering the identity of a softwareapplication (e.g., instant messaging tool) to be initiated by the OSupon the AMS application detecting a speech command using speechrecognition technology.

A macro can also represent recordable speech delivered by a microphonesingly or in combination with a headset for detection by anothersoftware application through speech recognition or for delivery of therecorded speech to other parties. In yet another embodiment a macro canrepresent recordable navigation of an accessory such as a joystick ofthe gaming controller 115, recordable selections of buttons of thegaming controller 115, and so on. Macros can also be combinations of theabove illustrations with selected actions from the Actions 130 menu.Macros can be created from the GUI 101 by selecting a “Record Macro”button 148. The macro can be given a name and category in user-definedfields 140 and 142.

Upon selecting the Record Macro button 148, a macro can be generated byselection of input functions on an accessory (e.g., Ctrl A, speech,navigation knob movements of the gaming controller 115, etc.) and/or bymanual entry in field 144 (e.g., typing the name and location of asoftware application to be initiated by an OS, such as an instantmessaging application, keyboard entries such as Ctrl A, etc.). Once themacro is created, it can be tested by selecting button 150 which canrepeat the sequence specified in field 144. The clone button 152 can beselected to replicate the macro sequence if desired. Fields 142 can alsopresent timing characteristics of the stimulation sequence in the macrowith the ability to modify and thereby customize the timing of one ormore stimulations in the stimulation sequence. Once the macro has beenfully defined, selection of button 154 records the macro in step 520.The recording step can be combined with a step for adding the macro tothe associable items Actions column 130, thereby providing the user themeans to associate the macro with input functions of the accessories(e.g., one or more keys of the keyboard 108, buttons of the gamingcontroller 115, etc.).

In step 522, the AMS application can respond to drag and dropassociations of actions with input functions of the keyboard 108 or thegaming controller 115. Associations can also be made based on the two orthree dimensional movements of the gaming controller 115. If user inputindicates that a user is performing an association, the AMS applicationcan proceed to step 524 where it can determine if a profile has beenidentified in step 512 to record the association(s) detected. If aprofile has been identified, the associations are recorded/stored in theprofile in step 526. If a profile has not been identified in step 512,the AMS application can create a profile in step 528 for recording thedetected associations. In the same step, the user can name the newlycreated profile as desired. The newly created profile can also beassociated with one or more gaming software applications in step 530 forfuture reference. The AMS application can also record in a profile instep 526 associations based on gaming venue states. In this embodimentthe same stimuli generated by the gaming controller 115 can result indifferent substitutions based on the gaming venue state detected by theAMS application.

Referring back to step 526, once the associations have been recorded ina profile, the AMS application can determine in step 532 which of theaccessories shown illustratively in FIGS. 1-3 are programmable andavailable for programming. If the AMS application detects that anaccessory (e.g., keyboard 108, gaming controller 115) is communicativelycoupled to the computing device 206 and determines that the accessory iscapable of performing stimulus substitutions locally, the AMSapplication can proceed to step 534 of FIG. 5 where it submits theprofile and its contents for storage in the accessory (e.g., the gamingcontroller 115 in FIGS. 2-3 ). Once the accessory (e.g., the gamingcontroller 115) is programmed with the profile, the accessory canperform stimuli substitutions according to the associations recorded bythe AMS application in the profile. Alternatively, the AMS applicationcan store the profile in the computing device 206 of FIGS. 2-3 andperform substitutions of stimuli supplied by the gaming controller 115according to associations recorded in the profile by the AMSapplication.

The GUI 101 of FIG. 1 presented by the AMS application can have otherfunctions. For example, the GUI 101 can present a layout of theaccessory (button 122), how the accessory is illuminated whenassociations between input functions and actions are made (button 124),and configuration options for the accessory (button 126). The AMSapplication can adapt the GUI 101 to present more than one functionalGUI page. For instance, by selecting button 102, the AMS application canadapt the GUI 101 to present a means to create macros and associateactions to accessory input functions as depicted in FIG. 1 . Selectingbutton 104 can cause the AMS application to adapt the GUI 101 to presentstatistics from stimulation information and/or gaming action resultscaptured by the AMS application as described in the subject disclosure.Selecting button 106 can also cause the AMS application to adapt the GUI101 to present promotional offers and software updates.

The steps of method 500 in whole or in part can be repeated until adesirable pattern is achieved of associations between stimulus signalsgenerated by accessories and substitute stimuli. It would be apparent toan artisan with ordinary skill in the art that there can be numerousother approaches to accomplish the embodiments described by method 500or variants thereof. These undisclosed approaches are contemplated bythe subject disclosure.

FIG. 6 depicts a method 600 for illustrating additional operations ofthe AMS application. In the configurations of FIGS. 2-3 , the AMSapplication can be operating in whole or in part from the gamingcontroller 115, a gaming console 206, a computer 206, or a remote server(not shown). For illustration purposes, it is assumed the AMSapplication operates from the gaming console 206. Method 600 can beginwith the AMS application establishing communications in steps 602 and604 between the gaming console 206 and a gaming accessory such as thegaming controller 115, and a headset 114 such as shown in FIG. 1 . Thesesteps can represent for example a user starting the AMS application fromthe gaming console 206 and/or the user inserting at a USB port of thegaming console 206 a connector of a USB cable tethered to the gamingcontroller 115, which invokes the AMS application. In step 606, thegaming controller 115 and/or headset 114 can in turn provide the AMSapplication one or more accessory ID's, or the user can provide by wayof a keyboard or the gaming controller 115 user identification. With theaccessory ID's, or user input the AMS application can identify in step608 a user account associated with the gaming controller 115 and/orheadset 114. In step 610, the AMS application can retrieve one or moreprofiles associated with the user account.

In step 612, the user can be presented by way of a display coupled tothe gaming console 206 profiles available to the user to choose from. Ifthe user makes a selection, the AMS application proceeds to step 614where it retrieves from the selected profiles the association(s) storedtherein. If a selection is not made, the AMS application can proceed tostep 616 where it can determine whether a software gaming application(e.g., video game) is operating from the gaming console 206 or whetherthe gaming console 206 is communicating with the software gamingapplication by way of a remote system communicatively coupled to thegaming console 206 (e.g., on-line gaming server(s) presenting, forexample, World of Warcraft™). If a gaming software application isdetected, the AMS application proceeds to step 617 where it retrieves aprofile that matches the gaming application detected and theassociation(s) contained in the profile. As noted earlier,association(s) can represent accessory stimulations, navigation, speech,the invocation of other software applications, macros or other suitableassociations that result in substitute stimulations. The accessorystimulations can be stimulations that are generated by the gamingcontroller 115, as well as stimulations from other accessories (e.g.,headset 114), or combinations thereof.

Once a profile and its contents have been retrieved in either of steps614 or step 617, the AMS application can proceed to step 719 of FIG. 7where it monitors for a change in a gaming venue state based on thepresentations made by the gaming application, or API messages suppliedby the gaming application. At the start of a game, for example, thegaming venue state can be determined immediately depending on the gamingoptions chosen by the gamer. The AMS application can determine thegaming venue state by tracking the gaming options chosen by a gamer,receiving an API instruction from the gaming application, or byperforming image processing on the video presentation generated by thegaming application. For example, the AMS application can detect that thegamer has directed an avatar to enter a tank. The AMS application canretrieve in step 719 associations for the gaming controller 115 forcontrolling the tank.

The AMS application can process movements of the gaming controller 115forwards, backwards, or sideways in two or three dimensions to controlthe tanks movement. Similarly, rotating the gaming controller 115 ortilting the gaming controller 115 forward can cause an accelerometer,gyro or magnetometer of the gaming controller 115 to providenavigational data to the AMS application which can be substituted withan action to cause the tank to turn and/or move forward. The profileretrieved by the AMS application can indicate that the greater theforward tilt of the gaming controller 115, the greater the speed of thetank should be moving forward. Similarly, a rear tilt can generatenavigation data that is substituted with a reverse motion and/ordeceleration of the forward motion to stop or slow down the tank. Athree-dimensional lift of the mouse can cause the tank to steeraccording to the three-dimensional navigation data provided by thegaming controller 115. For example, navigation data associated with acombination of a forward tilt and right bank of the gaming controller115 can be substituted by the AMS application to cause an increase inforward speed of the tank with a turn to the right determined by the AMSapplication according to a degree of banking of the gaming controller115 to the right. In the above embodiment, the three-dimensionalnavigation data allows a gamer to control any directional vector of thetank including speed, direction, acceleration and deceleration.

In another illustration, the AMS application can detect a new gamingvenue state as a result of the gamer directing the avatar to leave thetank and travel on foot. Once again, the AMS application retrieves instep 719 associations related to the gaming venue state. In thisembodiment, selection of buttons of the gaming controller 115 can beassociated by the AMS application with weaponry selection, firing,reloading and so on. The movement of the gaming controller 115 in two orthree dimensions can control the direction of the avatar and/orselection or use of weaponry. Once the gaming venue state is detected instep 719, the AMS application retrieves the associations related to thevenue state and can perform substitutions of stimuli generated by thegaming controller 115, and/or speech commands received by microphone ofthe headset 114.

In one embodiment, the AMS application can be configured in step 719 toretrieve a profile that provides substitute stimuli for replacingcertain stimuli generated by accessories. The associations recorded inthe profile can be venue independent. In another embodiment, the AMSapplication can retrieve a combination of profiles, where one or moreprofiles provide substitute stimuli that are venue dependent and one ormore other profiles provide substitute stimuli that are venueindependent.

The AMS application can monitor in step 720 stimulations generated bythe accessories coupled to the gaming console 206. The stimulations canbe generated by the gamer manipulating the gaming controller 115, and/orby generating speech commands detected by a microphone of the headset114. If a stimulation is detected at step 720, the AMS application candetermine in step 722 whether to forward the detected stimulation(s) toan Operating System (OS) of the gaming console 206 or the gamingapplication directly without substitutions. This determination can bemade by comparing the detected stimulation(s) to correspondingassociations in one or more profiles retrieved by the AMS application.If the detected stimulation(s) match the associations, then the AMSapplication proceeds to step 740 where it retrieves substitutestimulation(s) in the profile(s). In step 742, the AMS application cansubstitute the detected stimulation(s) with the substitute stimulationsin the profile(s).

In one embodiment, the AMS application can track in step 744 thesubstitute stimulations by updating the stimulations with a uniqueidentifier such as a globally unique identifier (GUID). In thisembodiment, the AMS application can also add a time stamp to eachsubstitute stimulation to track when the substitution was performed. Inanother embodiment, the AMS application can track each substitutestimulation according to its order of submission to the gamingapplication. For instance, sequence numbers can be generated for thesubstitute stimulations to track the order in which they were submittedto the gaming application. In this embodiment, the substitutestimulations do not need to be updated with sequence numbers oridentifiers so long as the order of gaming action results submitted bythe gaming application to the AMS application remain in the same orderas the substitute stimulations were originally submitted.

For example, if a first stimulation sent to the gaming application bythe AMS application is a command to shoot, and a second stimulation sentto the gaming application is a command to shoot again, then so long asthe gaming application provides a first a game action result for thefirst shot, followed by a game action result for the second shot, thenthe substitute stimulations will not require updating with sequencenumbers since the game action results are reported in the order that thestimulations were sent. If on the other hand, the game action resultscan be submitted out of order, then updating the stimulations withsequence numbers or another suitable identifier would be required toenable the AMS application to properly track and correlate stimulationsand corresponding gaming action results.

Referring back to step 722, if the detected stimulation(s) do not matchan association in the profile(s), then the AMS application proceeds toone of steps 744 or 746 in order to track the stimulations of theaccessory as described above. In another embodiment, tracking oforiginal stimulations or substitute stimulations can be bypassed byskipping steps 744 or 746 and proceeding to step 770 of FIG. 7B.

Once the stimulations received in step 720 have been substituted withother stimulations at step 742 responsive to a detected association ormaintained unchanged responsive to detecting no association withsubstitute stimuli, and (optionally) the AMS application has chosen aproper tracking methodology for correlating gaming action results withstimulations, the AMS application can proceed to step 770 of FIG. 7B.Referring to FIG. 7B, at step 770, the AMS application can obtain anidentification of an action to monitor during a gameplay associated witha game. The identification of the action may include a specification ofa sound volume level associated with a user (e.g., a gamer). Theidentification of the action may include a specification of a number ofuser inputs exceeding a threshold. The number of user inputs may includea number of messages that are submitted, an identification of a contentof the messages, an identification of an emoji, or a combinationthereof. The identification of an action may include a gaming actionprovided by the game—see FIGS. 8-9 and accompanying descriptions.

At step 772, the AMS application can store a representation of a slidingwindow of the gameplay in a first storage medium (e.g., first storagemedium 1272 of FIG. 12 ). The storage of step 772 may occur in real-timeduring the gameplay. The representation of the sliding window of thegameplay may include a video, an image, an audio track, or a combinationthereof. The first storage medium may include a buffer of a graphicscard, a random-access memory, or a combination thereof.

The sliding window may be of a substantially fixed duration, such thatthe sliding window progresses as the user/gamer continues to play agame. For example, and briefly referring to FIG. 11 , a sliding window1100 (as a function of time t) is shown. As gameplay progresses, anew/supplemental representation of the gameplay may be added as shownvia reference character/dashed portion 1102. In order to accommodatestorage of the portion 1102, another portion 1104 may bedeleted/overwritten. In the embodiment shown in FIG. 11 , the portion1104 to be deleted/overwritten corresponds to theoldest/earliest-in-time portion of the window 1100. In some embodiments,a portion other than, or in addition to, the oldest portion may beidentified for being deleted/overwritten. Still further, in someembodiments the sliding window 1100 may be of a variable duration. Forexample, the duration/length of the sliding window may be a function ofnetwork traffic, a capability of a device (e.g., storage capacity),user/gamer inputs, etc.

Referring back to FIG. 7B, at step 774, the AMS application can monitorfor the identification of the action during the gameplay.

At step 776, the AMS application can detect the identification duringthe gameplay responsive to the monitoring. In some embodiments, whetheran event has occurred or not, as reflected by the detection of step 776,may be based on a comparison of game action with one or more thresholds.Such thresholds may be specified by users/gamers (e.g., in accordancewith user inputs/preferences), may be predetermined based on one or morerules/configurations associated with a game, etc.

At step 778, the AMS application can store at least a portion of therepresentation of the sliding window of the gameplay in a second storagemedium. The second storage medium may be the same as, or different from,the first storage medium. The second storage medium may include a serverassociated with a social media platform, a server associated with avirtual machine, a memory contained within a common housing as the firststorage medium, a network element (e.g., a router, a gateway, a switch,etc.), or a combination thereof.

The storing of step 778 may include storing a video of a gamer, an imageof the gamer (e.g., a thumbnail or icon representation of the gamer), anaudio track of the gamer, or a combination thereof.

The storing of step 778 may include presenting a prompt (potentiallyresponsive to the monitoring of step 774), placing a copy of therepresentation of the sliding window of the gameplay in a third storagemedium, which may be different from the first storage medium and/or thesecond storage medium, receiving a user input in response to the prompt,and storing the copy in the second storage medium responsive to the userinput.

The placement of the representation/copy of the sliding window of thegameplay in the third storage medium may free/alleviate the firststorage medium, such that the first storage medium can continuecapturing gameplay/action as the gameplay continues subsequent to thedetection of step 776. Also, the placement of the representation/copy ofthe sliding window of the gameplay in the third storage medium may freethe user/gamer of not having to commit to placing therepresentation/copy of the sliding window of the gameplay into morepermanent storage (e.g., the second storage medium). For example,placement in the third storage medium may facilitate editing or reviewoperations of the representation/copy of the sliding window prior touploading the same to the second storage medium.

In some embodiments, the placing of the copy of the representation ofthe sliding window of the gameplay in the third storage medium mayinclude initiating a timer to store a second sliding window of therepresentation after detecting the action, thereby resulting in anupdated representation of the sliding window of the gameplay. Responsiveto detecting an expiration of the timer, the updated representation maybe stored in the third storage medium. A length of the timer may bebased on a user input.

In some embodiments, the storing of step 778 may include storing a newrepresentation of the sliding window of the gameplay in the firststorage medium during the gameplay after placing the copy in the thirdstorage medium; in some embodiments, the storage of the newrepresentation may coincide with a step that is separate from step 778.

At step 780, the AMS application may present (e.g., simultaneouslypresent) the representation of the sliding window and/or the video,image, and/or audio track of the gamer, or a combination thereof. Insome embodiments, a user/gamer may generate media that may be shared onone or more platforms (e.g., social media platforms) as a game isoccurring, where the media may include the representation of the slidingwindow and/or the video, image, and/or audio track of the gamer, or acombination thereof. Alternatively, the user/gamer may generate themedia following the conclusion of the game in order to avoiddistractions during the game.

One or more of the steps shown in conjunction with FIG. 7B may beexecuted more than once. For example, subsequent to storing therepresentation of the sliding window of the gameplay in the secondstorage medium as part of step 778, a second representation of thesliding window of the gameplay may be stored in the first storage medium(as part of a second execution of step 772). The storing of the secondrepresentation of the sliding window of the gameplay may overwrite atleast a portion of the representation of the sliding window of thegameplay in the first storage medium as described above.

FIG. 7C illustrates another embodiment of a method that may be executedin conjunction with the flow shown in FIG. 7A. As shown in FIG. 7C, instep 7170 the AMS application can obtain an identification of an actionto monitor during a gameplay associated with a game. The identificationof the action may include a specification of a number of actions perunit time.

In step 7172, the AMS application can store a representation of aportion of the gameplay in a storage medium.

In step 7174, the AMS application can monitor the gameplay for theidentification of the action.

In step 7176, the AMS application can apply a tag to the representationof the portion of the gameplay in the storage medium responsive to themonitoring.

The representation of the portion of the gameplay may include a firstvideo clip that occurs prior to an occurrence of the action and a secondvideo clip that occurs subsequent to the action. A first-time durationof the first video clip, a first resolution of the first video clip, asecond time duration of the second video clip, and a second resolutionof the second video clip may be based on one or more user preferences,network traffic, a device capability, etc.

The representation of the portion of the gameplay may include a videoclip. The tag may include a watermark that is applied to the video clip.The watermark may include the identification of the action. The tag mayinclude metadata that is associated with the video clip. The metadatamay be searchable via a search engine. The metadata may include aselectable link that, when selected, causes a client device to obtainthe video clip.

FIG. 7D illustrates another embodiment of a method that may be executedin conjunction with the flow shown in FIG. 7A. As shown in FIG. 7D, instep 7270 the AMS application can monitor for an identification of anaction during a gameplay.

In step 7272, the AMS application can detect the identification of theaction during the gameplay responsive to the monitoring.

In step 7274, the AMS application can present a prompt responsive to thedetecting.

In step 7276, the AMS application can store a representation of aportion of the gameplay, a representation of a gamer controlling thegameplay, or a combination thereof, in a storage medium according to auser-generated input associated with the prompt.

In some embodiments, machine-learning/artificial intelligence may beapplied to identify portions of a gameplay that are memorable or are ofinterest to a user (e.g., a gamer). For example, responsive to theuser-generated input associated with the prompt in step 7276, the AMSapplication can monitor for a second identification of the action (or analternative action) in step 7278.

In step 7280, the AMS application can detect the second identificationof the action (or the alternative action) during the gameplay responsiveto the monitoring for the second identification.

In step 7282, the AMS application can store a second representation of asecond portion of the gameplay, a second representation of the gamer, ora combination thereof, in the storage medium without presenting a secondprompt.

Once the AMS application at step 748 supplies to the OS of the computingdevice 206 a gaming action (i.e., one or more stimulations) from themethod of FIG. 7B, the method of FIG. 7C, the method of FIG. 7D, or acombination thereof, the AMS application can proceed to step 734. Thegaming action supplied to the OS at step 748 can be the unadulterated“original” gaming action of step 720, or an alternative gaming actiongenerated by steps 744 or 746. At step 734, the OS determines whether toinvoke in step 736 a software application identified in thestimulation(s) (e.g., gamer says “turn on team chat”, which invokes achat application), whether to forward the received stimulation(s) to thegaming software application in step 738, or combinations thereof.

Contemporaneous to the embodiments described above, the AMS applicationcan monitor in step 750 for game action results supplied by the gamingapplication via API messages previously described. For instance, supposethe stimulation sent to the gaming application in step 738 is a commandto shoot a pistol. The gaming application can determine that the shotfired resulted in a miss of a target or a hit. The gaming applicationcan respond with a message which is submitted by way of the API to theAMS application that indicates the shot fired resulted in a miss or ahit. If IDs such as GUIDs were sent with each stimulation, the gamingapplication can submit game action results with their corresponding GUIDto enable the AMS application to correlate the gaming action resultswith stimulations having the same GUID.

For example, if the command to shoot included the ID “1234”, then thegame action result indicating a miss will include the ID “1234”,enabling the AMS application in step 752 to correlate the game actionresult to the stimulation having the same ID. If on other hand, theorder of game action results can be maintained consistent with the orderof the stimulations, then the AMS application can correlate in step 754stimulations with game action results by the order in which stimulationwere submitted and the order in which game action results are received.In step 756, the AMS application can catalogue stimulations and gameaction results. In another embodiment, the AMS application can beadapted to catalogue the stimulations in step 760. In this embodiment,step 760 can be performed as an alternative to steps 750 through 756. Inanother embodiment, step 760 can be performed in combination with steps750 through 756 in order to generate a catalogue of stimulations, and acatalogue for gaming action results correlated to the stimulations.

FIG. 7E illustrates an interface that may be used to present at least aportion of a gameplay associated with a game. Various controls/commands,such as for example VCR types/styles of controls/commands, may bepresented as a part of the interface to facilitate a recording orcapture of one or more portions of the gameplay.

FIG. 7F illustrates an interface that may provide control over arecording or sharing of one or more representations (e.g., clips) of agameplay associated with a game. Various controls, such as for example a“share” button or the like, may be provided to enable a user (e.g., agamer) to post or otherwise share the representation(s). In someembodiments, editing controls may be provided to allow the user tocustomize the representation prior to, or subsequent to, sharing therepresentation.

In some embodiments, a user/gamer may have an ability to supplement therepresentation of the gameplay with commentary that describes, forexample, what the user's thought process was during thecaptured/represented portion of the gameplay. In this respect, andassuming that the user/gamer is viewed or otherwise characterized as anexpert in the game, a sharing of the representation of the gameplay mayserve as a tutorial for novice users.

FIG. 7G illustrates an interface that may present a tag 702 g (e.g., awatermark and/or metadata) associated with a representation of agameplay. The tag 702 g may include data acquired/obtained during thegameplay, such as for example a statement or other indication of resultsobtained by the gamer during the gameplay. Such a statement or otherindication may be received via, e.g., a microphone, a keyboard, a mobiledevice, a computing/gaming console, etc.

The methods described herein (e.g., the methods described above inconjunction with FIGS. 7A-7D) may incorporate additional aspects. Forexample, in some embodiments a clip may be generated based on a userdefined keybind (on a keyboard, mouse, or controller). Keybinds totrigger the clipping of a buffer to save to a local file system may becustomized (e.g., may be based on user preferences). The gamer will beable to choose: the actual key to bind to the action, and the time sliceto save (N seconds before and N′ seconds after).

In some embodiments, clips may be auto-generated based on some event,such as for example a detected event, an audible input (e.g.,screaming), messages associated with a chat client, etc. In someembodiments, default settings may be provided, and those settings may beat least partially overridden/replaced based on affirmative user inputsand/or based on artificial intelligence/machine-learned userpreferences.

In some embodiments, one or more filtering techniques may be applied toremove content from a representation of a gameplay that is not ofinterest. Such filtering may be based on one or more userinputs/preferences, may be learned over time via machinelearning/artificial intelligence, etc. If multiple events/actions thatare being monitored for happen within a threshold amount of time (whichmay coincide with a buffer time), an event/action endpoint may beextended to create one long time slice/representation of the gameplay.Alternatively, separate representations may be generated in someembodiments.

In some embodiments, tagging (e.g., watermarking) may be overlaid on arepresentation (e.g., a video) of a gameplay. A watermark may have agiven level of transparency associated with it to avoidobscuring/blocking the representation of the gameplay. One or more logosmay be applied as part of the tagging. In some embodiments, a watermarkmay pulsate or otherwise fade in-and-out. In this respect, dynamicwatermarks may be used. The use of a dynamic watermark may serve to drawadditional additional/incremental attention to the watermark, which maybe useful for promotional/marketing/branding purposes

Aspects of sharing the representation of the gameplay may be controlledvia one or more control parameters. Such control parameters maycondition the sharing on a size of the representation (e.g., a videolength), the content of the representation (e.g., controls may bepresent to limit a dissemination of the representation in view ofintellectual property rights or other rights), etc. In some embodiments,a sharing of the representation of the gameplay may be limited to usersthat the gamer (or other entity) authorizes. For example, the sharingmay be based on identifying a contact (e.g., a friend) of the gamer inone or more applications (e.g., a phone application, an emailapplication, a text message application, a social media application,etc.).

In some embodiments, audio from the game can be monitored and analyzedin real time or on the fly to generate game events. In addition, soundsthat occur during a game can be analyzed to generate notices to one ormore game participants; these notices can include live clips of thegameplay (sometimes referred to as moment clips), and/or statisticsrelated to one or more players' performance, equipment, health status,etc.

Referring to FIG. 7H, at step 7320, the AMS application can obtain oneor more audio profiles (e.g. spectral profiles) of sounds that may bereceived from a gaming software application at a player's station duringa gameplay associated with a game. The sounds may correspond to gameaction (for example, machine-gun fire, explosives detonating, etc.) orbe generated by the player (for example, the player saying specificwords during a game session). Game action sounds can be based on theplayer's actions (for example, the player shooting) or on anotherplayer's actions (for example, the other player detonating anexplosive). In an embodiment, the audio profiles are stored locally atthe player's station. In other embodiments, the audio profiles can bestored on a server or gaming console executing the gaming softwareapplication associated with the game.

The AMS application monitors sounds generated at the player's station(step 7322). In an embodiment, the AMS application can distinguishbetween sounds emitted by the gaming software application and presentedaudibly by the player's station and sounds produced in the environmentof the player. (For example, in player station or equipment 7411 of FIG.7J, if a sound 7419 detected via microphone 7414 has a similar profileand is in large part contemporaneous with a sound 7418 emitted fromspeaker 7413, the sound is understood as a game sound; otherwise, aplayer environment sound, which can be a sound produced by the player.)In a particular embodiment, the player station includes a spectralfilter that can be tuned to detect player environment sounds with knowncharacteristics.

When a sound is detected (step 7324), its spectral profile is analyzedand compared with the audio profiles. In an embodiment, the monitoringcan include a comparison to determine a correlation; the correlation canbe based on a threshold of similarity of characteristics of the detectedsound to characteristics of a reference sound in the audio profiles. Ifthe profile of the detected sound matches a stored profile (step 7326),a gaming event can be identified (e.g., explosion detected). In anembodiment, the gaming event can be distributed as a noticecorresponding to the detected sound that is distributed to otherparticipants in the game session (step 7328). In additional embodiments,the gaming event can trigger detection of an image on the display 7415of the player station to further analyze the significance of the gamingevent and generate further information that may be useful to otherplayers (e.g., explosion detected, but health of player near explosionhas not been affected). Such enhanced information can be conveyed in thenotice to other players. In further embodiments, the notice can betriggered on detection of a displayed image and/or a game-producedsound, in combination with a player's speech, which can be conveyed withthe notice. In other embodiments, the AMS the gaming event can cause theAMS to generate a gaming action that can be sent to the game and/or theplayers (e.g., initiate shield protection for affected player, sendmessage to teammates that player's shield has been initiated due to anexplosion). The gaming action can be pre-stored and associated with thedetected sound and/or image recognition at the time of the sound via theAMS application interface 100 of FIG. 1 .

The notice referred to early can take several forms, in accordance withembodiments of the disclosure. The AMS application can send a textmessage such as “Player A taking fire” or “Bomb detonated at Player A.”In an embodiment, this message can be shown on the display of the playerstation, superimposed on the gaming video being presented; in anotherembodiment, the message can be sent to a different player device. TheAMS application can also analyze the current state of the game action asseen by the player, and provide additional information (e.g.,statistics) regarding the game action associated with the detectedsound; for example, “Bomb detonated near player A, player A's health hasdiminished by x, bomb type is b, player A is engaged with n opponents.”

In additional embodiments, the gaming event detected by sound (orcombination of sound and imaging) can trigger a recording of a videoclip before and after the gaming event as described in the embodimentsof FIGS. 7A-7G. The notice can be included in the video cliprepresenting the gameplay occurring at the player's station at the timethe sound is detected (that is, a live clip with respect to the detectedsound). In some embodiments, the live clip is automatically generatedwhen the sound is detected. The AMS application can distribute the liveclip as part of the notice (for example, text “bomb near player A”superimposed on the live clip). In an embodiment, the AMS applicationgenerates the text notice by performing image processing of the liveclip.

The notice can also include a sequence of two or more video clips (forexample, a live clip combined with a clip of game action from 30 secondsearlier). In an embodiment, the triggering sound and/or image isanalyzed to determine which video clips are to be combined anddistributed to other participants.

In a further embodiment, the AMS application shares the player's liveclip (and/or additional clips in a sequence) with another player uponrequest by that player. Alternatively, the AMS application can selectone or more other players for sharing the video clip(s), based onanalysis of the triggering sound (or combination of sound/image/speech)and statistics regarding the player and other players. For example, asound can trigger analysis of the player's health status, equipmentstatus (e.g. remaining ammunition) and location, correlated withcorresponding data regarding other players; the notice can then be sentto the closest player having resources that can be shared.

In an additional embodiment, the AMS application can analyze the gameaction shown in the live clip together with recent movements by theplayer's avatar, and provide recommendations associated with thedetected gaming action (for example, “Run east 100 yards”).

Referring to FIG. 7I, the AMS application can maintain a log of soundsdetected during a game session. If a detected sound does not have aprofile matching the stored audio profiles (step 7376/N) but hasrecurred during the game session (step 7382/Y), the AMS application canadd the profile for that sound to the stored audio profiles (step 7384).In an embodiment, the recurring sound has its profile added after it hasbeen detected a prescribed number of times. The AMS application cananalyze the clips corresponding to the recurring sound to identify thesound, so that the profile of the sound can be stored with a text labelidentifying the sound.

FIGS. 7J and 7K schematically illustrate a system operating at least inpart according to the methods of FIGS. 5, 6, and 7A-7C. As shown in FIG.7J, a player 7410 uses equipment 7411 (generally referred to as a playerstation), which includes a local processing system, for playing a game;in this embodiment, AMS application 7412 and game application 7425execute on the player station. Game application 7425 can be downloadedfrom a game server 7405 via a network 7420. Player 7410 can engage withthe game action using any of several accessories including, for example,a mouse 7416.

In this embodiment, the player station stores audio profiles 7430 forgame-action sounds (e.g. sound 7418 from speaker 7413) andplayer-generated sounds 7419 (e.g. words spoken by the player) detectedby microphone 7414. As shown in FIG. 7K, a notice 7510 regarding adetected sound can be distributed via network 7420 to equipment of otherplayers 7511, 7512, 7513 participating in the game session. In anembodiment, the notice includes a live clip associated with occurrenceof the sound. In a further embodiment, the live clip is sent toequipment of a player in response to a request (e.g. request 7515 fromequipment of player 7511).

FIG. 7L depicts an illustrative embodiment of a block diagram for anaudio processing system 7600 for a gaming system in in accordance withaspects of the subject disclosure. The audio processing system 7600 maycooperate with a gaming system to provide audio mixing and equalizationfor multiple audio streams associated with the gaming system. Inembodiments, the gaming system may be embodiment as gaming controller115 or computing device 206 (FIG. 2 , FIG. 3 ). The computing device 206operates according to an operating system (OS) for controlling variousfunctions of the computing device 206 including input of data and otherinformation and output of data or other information. The audioprocessing system 7600 further cooperates with the OS of the computingdevice to control audio processing, including mixing and equalization ofaudio streams associated with the computing device 206.

The audio processing system 7600 includes a plurality of audio inputchannels including, in this embodiment, a game audio input channel 7602,a chat audio input channel 7604 and a capture audio channel 7606.Further, the audio processing system 7600 includes one or more audiooutput devices 7608, a graphical user interface (GUI) 7610 and remoteaudio devices 7612. Other embodiments may include additional oralternative features and functions.

The game audio input channel 7602 operates under control of the GUI 7610to select one or more audio streams from game audio source 7614 and tocontrol audio processing of the selected audio streams in the game audioprocessing module 7618. Further, the game audio input channel 7602operates under control of the GUI 7610 to route processed audio of thegame audio input channel 7602 to an output device 7608. The game audioinput channel 7602 includes a game audio source 7614, a gaming virtualaudio device 7616, and an audio processing module 7618. Otherembodiments may include other or alternative features or functions. Thegame audio source 7614 may be, for example, a gaming system, one or moreapplication programs running on a device such as computing device 206and other sources accessed by a user of the gaming system, such as anetwork source. In the illustrated embodiment, game audio source 7614may be an application or online source such as Steam® for gamingsoftware and applications (available online at steampowered.com) orLeague of Legends® or Counter-Strike:Global Offensive (CS GO)®. Theaudio from the gaming system may include game sounds associated withvideo displayed on a video display device of the computing device 206.In the case of a war simulation game, for example, the game audio inputstream 7602 may include sounds of gun shots, footsteps of teammates oropponents, explosions, vehicle noises, and others.

The game audio input stream 7602 may also include audio from othersources operating as game audio source 7614. Some applications generateaudio and may be active on a computing device 206 along with the gamingsystem. For example, a gamer may be listening to music while playing agame of the gaming system. The game audio input channel 7602 may includethe music audio stream from a music application. Further, an audiostream may be received from a remote location, over a network such asthe public internet. For example, the game may be watching a movieprovided by an online provider such as Netflix® or other source. Thegame audio input channel 7602 may include the audio received over thenetwork.

In some applications, the game audio input stream 7602 includesmultichannel audio or surround sound. Multichannel audio may includestereo audio with sound separation into right (R) and left (L) channels.Stereo audio may readily be provided to speakers built-in to a gamingdevice such as computing device 206. Stereo audio may also be readilyprovided to headphones worn by a gamer or other user.

Some applications such as games or videos produce surround sound audio.For example, surround sound audio may include techniques for enrichingthe fidelity and depth of sound reproduction by using multiple audiochannels from multiple speakers that surround the listener, referred toas surround channels. Surround sound audio may use four to seven or moreindependent audio channels and speakers placed in front of and behindthe listener in order to surround the listener with sound. Surroundsound audio may be produced according to a standard which specifiesencoding and decoding of audio data. One example of a surround soundstandard is referred to as 5.1 surround sound. The 5.1 standard callsfor six speakers including center (C) in front of the listener; frontleft (FL) and front right (FR) in front of the listener at angles ofthirty degrees off center to the listener; left surround (Ls) and rightsurround (Rs) at angles of 100 to 120 degrees off center to thelistener; and a subwoofer, the position of which is not specified. The5.1 surround sound standard is published as Recommendation ITU-RBS.775-3 (August 2012 by the International Telecommunications Union(ITU). Another example of surround sound is referred to as 7.1 surroundsound. 7.1 surround sound is similar to 5.1 surround sound but with foursurround channels instead of two, the rear left (RL) and rear right (RR)at angles of 135 to 150 degrees off center to the listener; the sideleft (SL) and side right (SR) at angles of 90 to 110 degrees off centerto the listener. The 7.1 surround sound standard is published as ReportITU-R BS.2159-7, February 2015, by the ITU.

Surround sound audio is typically directed to a listener position wherethe audio effects work best. The surround sound encoding presents afixed or forward perspective of the sound field to the listener at thislocation. The technique enhances the perception of sound spatializationby exploiting sound localization. Sound localization refers to alistener's ability to identify the location or origin of a detectedsound in direction and distance. In a surround sound system, soundlocalization is achieved by using multiple discrete audio channelsrouted to an array of loudspeakers. Each respective audio channel isrouted to a respective loudspeaker.

Multi-channel or surround sound audio can enhance the enjoyment for alistener. This is particularly true for a listener enjoying a video,film or audio such as a concert in a home theater experience. This isalso particularly true for a gamer participating in a game using agaming system on computing device 206.

The gaming virtual audio device 7616 provides device routing for thegame audio source 7614. The gaming virtual audio device 7616 may form anaudio logical device. The gaming virtual audio device 7616 may be adevice used for example for recording the received audio data or forsending the received audio data on a network. In contrast to a virtualaudio device, a real audio device is a physical device such as a laptopcomputer audio system or a Bluetooth speaker. On a computer such as alaptop computer forming the computing device 206, an operating systemwill expose a digital device. The operating system will see the digitaldevice as an addressable endpoint. There is no real, physical systembehind the addressable endpoint, including no hardware and no firmware.The gaming virtual audio device 7616 is merely a device in the sense ofthe operating system. The gaming virtual audio device 7616 may presentitself to the operating system of the computing device 206 as a soundcard or other physical device. It may merely be an address to whichaudio data is written by the processing system under control of theoperating system. The audio data may be subsequently read and processedby other devices. However, the gaming virtual audio device 7616 is avirtual device because there is no physical device associated with it.

In an embodiment, the gaming virtual audio device 7616 may be configuredas a portion of code stored in memory for performing suitable functionssuch as controlling a processing system. The gaming virtual audio device7616 may receive multiple audio streams of the game audio source 7614from the game input audio stream 7602 and combine them to form a targetaudio stream. The gaming virtual audio device 7616 is configured toroute the target audio stream to the audio processing module 7618.

In an embodiment, the gaming virtual audio device 7616 includes a sourcecode that embeds a functionality capable of routing the target audiostream to the audio processing module 7618. In practice, the gamingvirtual audio device 7616 re-emits the audio data received on its audioreceiver endpoint in the form of the plurality of first audio streams onan audio emitter endpoint that will be read by the audio processingmodule 7618. In some embodiments in conjunction with the Windowsoperating system the audio endpoint receiver of the gaming virtual audiodevice 7616 typically corresponds to the so-called Windows operatingsystem audio endpoint sink, and the audio endpoint emitter of the gamingvirtual audio device 7616 typically corresponds to the so-called Windowsoperating system audio endpoint source.

The audio processing module 7618 enables custom digital signalprocessing effects. In some embodiments, the audio processing module7618 may be embodied as a Windows audio processing object (APO). In theWindows® operating system, for example, audio processing objects (APOs)provide software based digital signal processing (DSP) for Windows audiostreams. An APO is a COM host object that contains an algorithm toprovide a specific DSP effect referred to as an audio effect. Examplesof APOs include graphic equalizers, reverb, tremolo, acoustic echocancellation (AEC) and automatic gain control (AGC). APOs are COM-based,real-time, in-process objects. The APO provides programmable audioprocessing.

The chat audio input channel 7604 includes one or more chat audiosources 7620, a chat virtual audio device 7621 and a chat neural networkeffects service 7622. The chat audio input channel 7604 operates undercontrol of the GUI 7610 to select one or more audio streams from thechat audio sources 7620 and to control audio processing of the selectedaudio streams in the chat neural network effects service 7622. Further,the chat audio input channel 7604 operates under control of the GUI 7610to route processed audio of the chat audio input channel 7604 to anoutput device 7608.

In accordance with various aspects described herein, a gamer usingcomputing device 206 to participate in a game or gaming application maycommunicate verbally with one or more teammates or others using a chatchannel. For example, the teammates may use a voice over internetprotocol (VoIP) application to communicate with each other. Examples ofsuitable applications include the Skype® application and the TeamSpeak®application. In other examples, the gaming application may include abuilt-in voice communication channel. The gamer or other user may accessthe GUI 7610 or the operating system of the computing device 206 toselect one or more chat audio sources 7620 for communication. Audio fromthe chat audio sources 7620 is provided to the audio processing system7600 at the chat audio input channel 7604 and may include voiceconversation by teammates of the gamer or other user associated with theaudio processing system 7600.

The chat virtual audio device 7621 operates on principles similar tothose described herein for the gaming audio device 7616. The chatvirtual audio device 7621 may be embodied as a module including softwarecode. The chat virtual audio device 7621 may operate in response tocontrol inputs from the GUI 7610 to select one or more of the more chataudio sources 7620. The chat virtual audio device 7621 routes an audiostream from the more chat audio sources 7620 to a selected destinationsuch as the neural network effects service 7622 in response to controldata from the GUI 7610. In some embodiments, multiple instances of thechat virtual audio device 7621 may be used to access and route multiplerespective audio streams from multiple respective chat audio sources7620.

The neural network effects service 7622 provides audio processing of theuser audio stream from the chat audio input channel 7604. In someembodiments, the neural network effects service 7624 implements anartificial intelligence routine for customizing one or more audiostreams such as the audio stream from the chat audio sources 7620. Forexample, the neural network effects service 7624 may implement anartificial intelligence-based noise cancellation for the user audiostream from the microphone 7623 that operates to remove background noiseoriginating in the environment of the user or gamer. The neural networkeffects service 7622 responds to control information from the GUI 7610to tailor audio processing in the audio stream received from the chatvirtual audio device 7621. FIGS. 7M-7T depict an illustrative embodimentof a GUI that may embody the GUI 7610. Further, the GUI 7610 providescontrol information to route audio stream from the neural networkeffects service 7622 to one or more output devices 7608.

The capture audio channel 7606 includes a microphone 7623, a neuralnetwork effects service 7624 and a microphone virtual audio device 7625.The capture audio channel 7606 corresponds to audio originating with agamer or user interacting locally with the gaming system on thecomputing device 206. For example, the user may interact with one ormore gaming accessories such as a gaming console to participate in thegame. As noted, the user may access other audio sources such as musicand video including audio from a local application or over a networksuch as the internet. As part of the interaction, the user providesspoken audio input which may be chat shared with gaming teammates, forexample. Chat or other conversation from the teammates is received atthe chat audio input channel 7604. Similarly, the reverse channel withchat or other conversation from the gamer to teammates is provided onthe capture audio channel 7606.

The microphone 7623 may include any microphone or other device forcapturing audio locally with the user. In an example, the user wearsheadphones or a headset equipped with a microphone. In another example,a laptop computer or other computing device 206 includes a microphonefor capturing user audio. The user audio stream from the microphone 7623is routed and processed according to the operating system of thecomputing device 206.

The neural network effects service 7624 provides audio processing of theuser audio stream from the microphone 7623. In some embodiments, theneural network effects service 7624 implements an artificialintelligence routine for customizing one or more audio streams such asthe user audio stream. For example, the neural network effects service7624 may implement an artificial intelligence-based noise cancellationfor the user audio stream from the microphone 7623 that operates toremove background noise originating in the environment of the user orgamer. Further, the neural network effects service 7624 responds tocontrol information from the GUI 7610 to tailor audio processing in theuser audio stream received from the microphone 7623. FIGS. 7M-7T depictan illustrative embodiment of a GUI that may embody the GUI 7610.

The user may wish to adjust how the user sounds on team chat toteammates or others who may hear the user The GUI 7610 may be used toadjust processing by the neural network effects service 7624 or otheraudio processing system to adjust how the user sounds to others. Forexample, some users want to have their voice sound a little deeper toothers. Some users may want their vice to sound a little clearer toothers, such as by adding noise reduction to the microphone audiostream. In another example, a user may select a pre-set audio processingconfiguration such as a broadcaster pre-set, which modifies the user'svoice to sound like a professional broadcast audio stream. In yet otherexamples, the user's voice may be completely modified to sounddifferent, such as a male user who wishes to sound female or a veryyoung user who wishes to sound very old. In another example, a femaleuser may modify her voice to sound male to avoid harassment that canoccur online. The user may choose to add effects such as compression toremove relatively high frequencies or relatively low frequencies.Moreover, the user may choose to apply parametric equalization to theuser's voice audio stream to further tailor the user's output audio.This may improve intelligibility of the user's voice. In anotherexample, the user may add an accent or remove an accent, such as theuser who adds an Australian accent to sound more charming or intelligentto American teammates. As indicated, the user may be provided with anumber of selectable pre-set audio modifications that may be applied bythe neural network effects service 7624. The selectable pre-set audiomodifications may be accessed and selected by actuation of the GUI 7610.

The microphone virtual audio device 7625 operates similarly to thosedescribed herein for the gaming virtual audio device 7616. Themicrophone virtual audio device 7625 may be embodied as a moduleincluding software code. The microphone virtual audio device 7625 mayoperate in response to control inputs from the GUI 7610 to select one ormore of the more destinations for an audio stream from the neuralnetwork effects service 7624.

Thus, the GUI 7610 provides control information to route an audio streamfrom the neural network effects service 7622 to remote audio devices7612. In the example, the remote audio devices 7612 may include any ofthe chat audio sources 7620, such as a VoIP application. Examples ofsuitable VoIP applications include the Skype® application and theTeamSpeak® application. Others may be specified, as well.

The audio processing system 7600 may create virtual audio devices suchas the gaming virtual audio device 7616, the chat virtual audio device7621 and the microphone virtual audio device 7625. Such virtual audiodevices may be accessed in the Windows operating system. Applicationscan be assigned to the virtual audio devices. The virtual audio devicemay be assigned names or other designations and may be actively selectedby the user or gamer. In the example of FIG. 7L, a single virtual audiodevice 7616 is illustrated for processing audio from games such as Steamas well as audio streams from video sources such as the Twitch® livevideo streaming service or a sports video program as well as audiostreams from audio sources such as the Spotify® audio streaming service.In other embodiments, a respective virtual audio device may beestablished for each respective audio stream handled by the audioprocessing system 7600.

In embodiments, the audio processing system can be running on a devicesuch as computing device 206 and processing audio from some or all openapplications on the device. For example, if five applications arecurrent running on the device. If three applications of the five openapplications are routed to the same virtual audio device 7616 and audioprocessing module 7618, the three applications will have the sameeffects applied, such as volume and equalization and filtering.Similarly, the other two applications, if routed to a different virtualaudio device analogous to virtual audio device 7616 and a differentaudio processing module analogous to audio processing module 7618, theother two applications will have the same effects applied. Thus, thevirtual audio device 7616 and audio processing module 7618, andanalogous devices for other audio streams, may form a switching modulefor mapping or routing audio streams from applications or other sourcesto any audio destination such as audio output devices 7608, thegraphical user interface (GUI) 7610 and the remote audio devices 7612.

FIG. 7L-1 and FIG. 7L-2 illustrate Windows® operating system controls ofa user interface for selecting device preferences. In particular, FIG.7L-1 and FIG. 7L-2 illustrate control of audio routing within a Windowsdevice such as computing device 206. A user may access the WindowsControl Panel to control audio stream routing. In FIG. 7L-1 and FIG.7L-2 , various audio sources are illustrated including in this example,System sounds, Discord, Microsoft Teams, OBS 27.0.1 and Slack. Systemsounds refers to audio generated internally by the Windows operatingsystem. Discord, Microsoft Teams, Open Broadcaster Software (OBS) andSlack are application programs for sharing audio and video within thecomputing device 206 and between the computing device 206 and otherdevices over a network. For example, in a gaming environment, teammatesmay chat or communicate verbally using a feature of the game or they mayuse a third-party app such as Discord or Slack.

The Windows control panel of FIG. 7L-1 and FIG. 7L-2 allows the user tocontrol association of audio sources and audio destinations. The usermay select an input device and an output device for each application. Inthe example of FIG. 7L-1 , an audio stream from the Discord applicationis selected to come from a Default input. Further, in the example ofFIG. 7L-2 , an audio stream from the Discord application is selected forrouting to a destination designated as SoundStage chat capture device(SteelSeries SoundStage device). In other examples, the applicationslisted on the left of FIG. 7L-1 and FIG. 7L-2 may include otherapplications for music, streaming video, broadcast television and othersources.

The operating system of the computing device 206, in this example, theWindows® operating system, may thus be used to map audio streams fromaudio sources to audio destinations. The audio sources include games,applications, network locations and any other source of audio to thecomputing device. Mapping the audio streams may further include mappingor routing particular audio streams to a particular audio processingmodule 7618, such as an APO or neural network effects service 7622.

In some embodiments, the audio processing system 7600 may automaticallycharacterize audio events occurring in the game audio stream, the chataudio stream, the microphone audio stream or some combination of these.In some embodiments, machine learning techniques may be used tocharacterize or profile audio events in the audio stream to identify apredetermined audio event such as a footstep or a gunshot associatedwith another player. In an example, the live game audio stream may beprovided to a neural network. The neural network may be trained usingany suitable data such audio from other games or conversations. Thetraining data may be tagged, for example, to identify predeterminedaudio events such as a gunshot or a footstep. Different characteristicsmay be further trained and identified, such as a footstep on wetpavement or a gunshot with ricochet. Once trained and provided with thelive audio, the neural network or other processing module may produce anindication when the predetermined audio event has occurred. In someembodiments, the indication is a value corresponding to the probabilitythat the predetermined audio event has occurred. If the probabilityexceeds a threshold, such as 75 percent or 95 percent, the audioprocessing system may conclude that the predetermined event has beendetected.

In response to detecting the predetermined audio event, the audioprocessing system may take further steps to enhance or adjust the audiocorresponding to the predetermined audio event. In an example, the audioprocessing system may automatically adjust parametric equalizer toemphasize spectral components of the predetermined audio event. Forexample, if a footstep is detected, the spectrum may be adjusted in theparametric equalizer to emphasize to the listener, the player, the soundof the footstep. Further, if the player is using a headset or otheraudio equipment that provides directionality or other surround soundeffect, the audio may be automatically adjusted to emphasize thedirection of origin of the predetermined event. For example, if thefootstep in the game comes from behind the player, the audio of thefootstep in the surround sound arrangement will emphasize the directionso that the audio from behind is louder or clearer and the footstep ismore distinct. Other sounds from that area may be suppressed toemphasize the footprint. In a video game in which the player has anassociated avatar, the directionality of the sound may be processedaccording to the position and orientation of the avatar in the game.

In other embodiments, a graphical user interface or other visualinterface may be activated to provide the player with a visualindication of the detected predetermined audio event. For example, thegraphical user interface may flash text advising of “footstep behindyou,” “possible gunshot detected to your right.” The nature of thevisual indication may be based on the nature of the detected audio eventand the probability or likelihood of detection, plus any otherinformation that can be derived from the audio stream.

Further, in some embodiments, the GUI 7610 may be used to control theassociation of audio sources and audio destinations. FIG. 7M and FIG.7M-1 illustrate an example in which the GUI 7610 of the audio processingsystem 7600 may be used to access the Windows Control panel to manageaudio stream configurations. In the illustrated embodiment, the WindowsControl panel image is produced on a display screen which displays theGUI 7610. In other embodiments, a custom graphical user interfacedisplay (not shown) may be produced with suitable controls enabling auser to access and modify mapping or routing of particular audiostreams. Any suitable interface controls may be provided includingmouse-based controls, touch-based controls on a touch-sensitive screenand voice-based controls.

In some embodiment, the audio processing system 7600 may automaticallydetect one or more audio sources and one or more audio destinations andautomatically route audio streams from source to destination. In someembodiments, the audio processing system 7600 may pre-populate a routingmatrix which can then by modified manually by the user, such as byaccessing one or more pop-up menus similar to those illustrated in FIG.7L-1 and FIG. 7L-2 or similar to those illustrated in FIG. 7M-1 or 7M-2. The audio processing system 7600 may select a group of audioapplications known to be the most popular applications and pre-populatethe menus with those applications. For example, the most popular voiceover internet protocol (VoIP) applications may be selected as defaults.In another example, if the user is known to have a preference for anapplication, the audio processing system 7600 may select the preferredapplication as a default. The user can access the GUI 7610 and modifythe selection as desired. The default selections for audio routing andaudio processing are automatically selected for the user upon start-upof the gaming system or other applications to simplify and streamlinethe process for the user. The user starts the game and his preferredaudio settings are established automatically. Moreover, if the user hasdifferent preferences for different games or different types of games,or for different types of applications such as music, video andstreaming video, the audio processing system 7600 automatically detectsthe type of game or type of application and selects the user's preferredsettings for each audio stream. Information defining the user'spreferred settings may be stored at any convenient location such aslocally at the computing device 206 or remotely at a server or otherstorage location accessible over a network such as the public internet.The user's preferred setting may be part of a profile or configurationfile of the user.

In some examples, the audio processing system 7600 may apply a headrelated transfer function (HRTF). HRTF is a methodology used to generatevirtual surround sound. It may be used to simulate virtual objects in avirtual surround environment using two speakers. Human soundlocalization may depend on the HRTF in the frequency domain and ahead-related Impulse Response (HRIR) in the time domain. The HRTFrelates to an acoustical footprint from a location in space to the earof a person. A soundwave traveling from a location in space to the earsof a person will be modified due to acoustical reflection anddiffraction phenomena by the listener's body, shoulders, head and earpinnae. The HRTFs contain the so-called localization cues, i.e. theinformation used by the human brain to decode the direction of a soundevent. Hearing a sound is affect by interaural time differences (ITD)and interaural level differences (ILD). These are mainly due to anacoustical head-shadowing effect. They enable lateral localization(left/right) by the listener, but they do not provide enough informationto precisely locate a sound source.

The HRTF includes various spectral cues used by the user to localizesound. These include notches and peaks in the audio spectrum. These areprimarily due to reflection on the listener's shoulders (low frequency)and reflection or diffraction by the ear pinnae of the listener (highfrequency). Such spectral cues provide information used by the brain ofthe listener to decode the front/back and up/down location of a soundsource.

Spatial sound over headphones uses HRTFs to simulate a sound source at aprecise location around the listener. Most conventional spatial sound orvirtual surround sound solutions use one unique generic HRTF profile,developed from one unique person or a dummy head. In an example, if S(f)is the signal of the sound source, the stereo signal playback by theheadphones is obtained by filtering it with the appropriate left HRTFand right HRTF. This may be described as follows:

S _(L)(f)=S(f)*HRTF_(L)(f,direction person)

S _(R)(f)=S(f)*HRTF_(R)(f,direction person)

for the left (L) ear or channel and right (R) ear or channel,respectively.

A generic HRTF profile cannot fit every listener, the localization cueswill be the right ones for some listeners, but not for others. Thelocalization cues are dependent on the anatomy of the listener. Thetypical drawback of using a generic HRTF profile is that the front/backand up/down localization is distorted. For example, sounds intended tobe heard in the front of the listener are heard by the listener in therear of the listener. Acquiring a personal HRTF for a user iscumbersome, expensive and requires substantial time and equipment, andis not realistic in most instances. Some conventional approaches to HRTFcustomization include use of pictures of the listener's ears to derivean adapted HRTF profile or allowing a user to select among tweakedversions of a single HRTF profile.

FIG. 7L-3 illustrates a process for listener selection of a HRTF profilefor use by the user. The technique of FIG. 7L-3 employs a version of A/Btesting to allow the user to select an HRTF that sounds best to thelistener. In the process illustrated in FIG. 7L-3 , several HRTFprofiles are presented to the user. The user is guided through a seriesof A/B comparison tests to select the best or most appropriate HRTF forthat user. The HRTF is then applied to an audio processing system suchas the audio processing system 7600 of FIG. 7L.

In the example embodiment, the listener is presented with a series ofeight HRTF profiles. Eight profiles are arbitrarily chosen. Any suitablenumber can be used. In FIG. 7L-3 , the HRTF profiles are labelled A1,B1, A2, B2, A3, B3, A4 and B4. Each HRTF proposal in the example isbased on different a HRTF developed for another person and is related todifferent morphologies, or persons with varying anatomy that will affectthe HRTF. In the example embodiment, the eight HRTF profiles arerepresentative of a large variety of morphologies or persons.

The process of FIG. 7L-3 is a guided selection by the user of a best oroptimal or closest-fit HRTF profile. The process may be described as atournament of several A/B listening tests. Thus, in the first round 7626of the tournament, the user is presented with a listening test andcomparison between HRTF profile A1 and HRTF profile B1. Further, in thefirst round 7626, the user is presented with a listening test andcomparison between HRTF profile A2 and HRTF profile B2. Still further inthe first round 7626, the user is presented with a listening test andcomparison between HRTF profile A3 and HRTF profile B3 and a listeningtest and comparison between HRTF profile A4 and HRTF profile B4.

Any suitable listening test may be provided. In some embodiments, audiotest files are presented to the listener that highlight commonperceptive drawbacks such as front/back discrimination difficulties. Inthe example, the listener first hears one or more audio test files asmodified by HRTF profile A1 and the same one or more audio test files asmodified by HRTF profile B1. The user selects or indicates whetherprofile A1 or profile B1 sounded better or more realistic or morenatural to the listener, focusing preferably on the ability of thelistener to spatially locate sounds played in the one or more audio testfiles. The selected profile, A1 or B1, is judged a winner of the firstround 7626. The user continues with the other A/B tests of the firstround, including profile A2 versus profile B2; profile A3 versus profileB3; and profile A4 versus profile B4. For each A/B test of the firstround 7626, a winner advances to the second round 7627.

The A/B testing process is repeated for the second round 7627. The fourwinners from the first round 7626 are subjected to two A/B tests and twowinners of the second round 7627 are advanced to the final round. Again,the winner of each A/B test is the HRTF profile that is judged best bythe listener for that listener, focusing on spatial location of soundsof the audio test files. A final round 7628 is conducted with a similarA/B test and an ultimate winner HRTF profile is selected as the best oroptimal HRTF profile for the listener among the eight HRTF profiles thatwere initially selected.

Throughout the process illustrated in FIG. 7L-3 , the user may interactwith a graphical user interface such as the GUI 7610 of FIG. 7L. Thegraphical user interface may provide written or audible instructions tothe listener to lead the listener through each A/B test and through eachround including the first round 7626, the second round 7627 and thefinal round 7628. Directions and guidance may be as detailed asnecessary for the listener. Generally, the process of FIG. 7L-3 needs tobe performed a single time to select an optimal HRTF profile for thelistener.

Using a tournament format for selecting an optimal HRTF profile for thelistener simplifies a process of selecting a best-fit or optimal HRTFprofile for the listener. The tournament format reduces the number ofA/B listening tests required of the user to identify the best fit forthis listener. The listening tests allow the listener to make an HRTFprofile selection that is the best fit for that listener, by directlycomparing with another HRTF profile. As the tournament advances throughthe first round 7626, the second round 7627 and the final round 7628,the HRTF profiles should generally be closer to the actual ideal HRTFprofile for this particular listener. The winner of the final round 7628is unlikely to be precisely the ideal HRTF profile for this listener butis likely to be a good fit and the best fit from a wide selection ofeight pre-selected HRTF profiles.

Once an HRTF profile for a user has been established, such as by theprocess illustrated in FIG. 7L-3 , the HRTF profile may be used in theaudio processing system 7600 of FIG. 7L. In some embodiments, the gameaudio input channel 7602 includes audio processing to establish HRTFprocessing. HRTF processing may be applied, for example to audio streamsencoded with 5.1 surround sound or 7.1 surround sound to provide thevirtual surround sound effect when the user listens through two stereospeakers. For example, the audio processing module 7618 may include anHRTF module operative to enable HRTF processing for the audio streamreceived from game audio, movie or video audio, and other streams tocreate a virtual surround sound effect for the user. The HRTF processingmay be applied after audio processing including parametric equalizationand other functions. The user may activate and control the HRTF moduleand HRTF processing using the GUI 7610 or in any other suitable manner.

FIGS. 7M-7T depict an illustrative embodiment of a graphical userinterface display of an audio processing system for a gaming system inaccordance with aspects of the subject disclosure. A user of the audioprocessing system 7600 of FIG. 7L can interact with the graphical userinterface to control aspects of the audio processing system 7600. In theexemplary embodiment, the user may participate in gameplay with thegaming system with other players including teammates. For example, theuser may interact with a personal computer, laptop computer or otherdevice that implements the gaming system and is in data communicationwith a network such the public internet. The gaming system generates anaudio stream of gaming audio that is presented to the user through anaudio output system such as headphones. The player and the teammates ofthe player converse together in an audio stream referred to as gamingchat. The gaming chat audio stream is received by the user's computerand is presented to the user through an audio output system such asheadphones. The user generates an audio stream by speaking into amicrophone, for example as part of the gaming chat. The speech of theuser is converted to data by the microphone and communicated over thenetwork to teammates of the user.

FIG. 7M depicts an illustrative embodiment of a mixer panel 7630 of thegraphical user interface. The mixer panel 7630 includes an audio streamselector 7631, a series of audio stream controllers 7632 and a series ofoutput selectors 7633. Other embodiments may include other features inaddition to those illustrated in FIG. 7M.

The audio processing system of FIG. 7L is configured to process threeseparate audio streams, Game, Chat, Microphone. Each audio stream isprocessed independently from the others. In the mixer panel 7630, theuser can assign a different audio device, such as a headset, speakers,microphone, or others, to each respective audio stream and tune finelythe volume of the respective audio stream. Furthermore, the user alsohas a master volume. Separately processing the respective audio streamsallows the user to finely tune the game sound without impactingteammates' voice audio on the chat audio stream.

The audio stream selector 7631 allows the user to select and control anaudio stream. The audio stream selector 7631 may be used to select fordisplay the mixer panel 7630, a gaming audio control panel, a chat audiocontrol panel and a microphone audio control panel. The audio streamselector 7631 may be used, for example to select using a mouse or otheraccessory, or by touching a touch-sensitive screen, to select aparticular audio stream. In the illustrated embodiment, the mixer panel7630 enables volume control over three separate audio streams of thegaming system, including a gaming audio stream, a team chat audio streamand a user's microphone audio stream, as well as a master audio volume.The user may select a particular audio stream and, using a mouse forexample, may drag the volume slider control for the selected audiostream to a desired level. In this manner, the relative volume of eachseparate audio stream may be tailored to the user's preferences or needsat a given time. The master volume slider allows the user to controloverall game audio volume from all sources.

The output selectors 7633 are embodied as a pop-up menu that may beselected by the user with a mouse or other accessory. The pop-up menu ispopulated by the system with the audio devices that the operating systemof the computer has identified. FIG. 7M-1 shows an example of selectingan audio output device on a pop-up menu. Generally, recognized audiodevices include speaker systems and microphone systems that are detectedby the operating system.

One of the recognized audio devices is designated as the default audiodevices.

By default, the Windows® operating system has a default multimediadevice. The default multimedia device is the stream used for gaming. Thedropdown menu 7633 may be used to control where audio output isdirected. To select chat, the user can use the menu to select where tousers wants to redirect the audio, such as headphones, speakers, audiodriver. On the dropdown menu 7633, the list is populated from knownwindows audio devices. Windows has a default audio communication device.The audio processing system sets Chat as the Windows default audiocommunication device and all the audio from the Chat audio stream getsrouted to the audio selected in the dropdown. In this example, the userdoes not choose where the audio comes from but chooses where y the audiois redirected to. The game itself outputs audio to the default Windowssetting, which may be set by the Windows operating system control panel.

FIG. 7M-2 shows an example of selecting sound input and sound outputoptions in a device with a Windows® operating system. Other devices maybe selected through actuation of the pop-up menu of the output selectors7633. In the illustrated example of FIG. 7M, an Arctis 7 Plus headphonesystem is selected as the output device for the gaming audio stream, thechat audio stream and the microphone audio stream. Using the outputselectors 7633, the user may route the audio streams to different outputdevices. FIG. 7M-3 illustrates mixing of different audio levels for thegaming audio stream, the chat audio stream and the player's microphoneaudio stream. FIG. 7N-1 illustrates

Thus, the mixer panel 7630 enables separate audio processing for each ofthe game audio stream, the chat audio stream and the microphone audiostream. By interacting with the graphical user interface including themixer panel 7630, the user can independently control the volume andaudio effects of each of these streams.

FIG. 7N depicts an illustrative embodiment of a parametric equalizer7635 of the graphical user interface. An equalizer is a tool by whichthe user can adjust and control the sound to improve audio duringgameplay or other situations. The equalizer allows the user to hear moreor less of the sound from a specific frequency range. Adjusting eachfrequency lets the user hear more of what the user wants to hear andless of what does not want to hear.

The parametric equalizer 7635 provides an interactive but easy to use,professional level 10-band parametric equalizer (EQ) for completecustomization on the user's gaming audio stream. The gaming audio streamincludes the audio stream of game sounds generated by the gaming systemto accompany game visuals. For example, the user can lower the sound ofexplosions produced by the game and increase the important sounds offootsteps and reloading produced by the game so that the user can betterunderstand activities of opponents in game. The parametric equalizerallows the user to boost or cut the sound over custom frequency bandsaccording to a set of parameters, giving precise control. In particular,for a selected frequency band, the user can select the center frequency,the amplitude, the bandwidth and the filter type. Any tuning will beapplied to the user's sound and visible on the graph in real time. Insome embodiments, the tuning will affect only the game audio stream.Other audio streams won't be affected.

The graphical user interface allows the user to interact with thedisplay on a display screen of the user's device using a mouse or otheraccessory. In the parametric equalizer 7635, the center frequency can beshifted by the user to any value between 20 Hz and 20 kHz by dragging onthe graph one of the available band dots or by selecting a band (from 1to 10) and then using the frequency slider. Once selected, the amplitudeof the band can be tuned up to +12 dB or down to −12 dB directly on thegraph or with the gain slider.

In the graphical user interface illustrated in FIG. 7N, the user mayselect one of the dots corresponding to a frequency band or filter andadjust the gain applied by a respective filter by dragging the dot up ordown vertically on the display screen. A value higher on the screencorresponds to a higher gain for that filter. The continuous line in thegraphical user interface shows the actual frequencies that are beingapplied or curves out between the dots. Selecting the dots allows theuser to move the line. Curve smoothing is applied between adjacent dotsto create a smooth frequency curve. FIG. 7N-1 illustrates an example ofadjusting the parametric equalizer 7635 for the chat audio stream toimprove the intelligibility of teammates speech conveyed to the userover the chat audio stream.

In the illustrated embodiment, the parametric equalizer 7635 includes aparameter selector 7636, an equalization selector 7637 and a filter typeselector 7638. The parametric equalizer 7635 implements a series offilters for controlling audio processing. Using the features of theparametric equalizer 7635, the user can independently adjust the gain,frequency, quality (Q) factor and apply audio filters for up to 10 datapoints or frequency bands to customize the game audio.

The parameter selector 7636 enables the user to select a parameter tocontrol on the graphical user interface. The user interface is dividedinto a series of frequency bands of the audible spectrum. In the exampleof FIG. 7N, the frequency bands are centered at 20 Hz, 50 Hz, 100 Hz,200 Hz, 500 Hz, 1 kHz, 2 kHz, 5 kHz, 10 kHz and 20 kHz. These centerfrequencies are exemplary only. Center frequencies may be selected toemphasize certain regions of the audible spectrum, such as relativelylow frequencies. The selection may be based on the particular audiocontent of the game being played or some aspect of the game beingplayed. For example, in a game involving players walking and shooting,relatively low frequency sounds carry substantial information that isuseful to the gamer to understand unseen action of the game.

In the illustrated embodiment, the user can separately control the gainof a selected frequency band. Each respective frequency band correspondsto respective filter of the series of filters that form the parametricequalizer 7635. In the illustrated example, a frequency slider isactuated to select a frequency at 32.03 Hz and a gain slider is selectedto adjust the gain at 32.03 Hz to a value of 2.1 dB. Moreover, a Qfactor slider may be actuated to adjust the Q factor of each respectivefilter. The Q factor slider adjusts the Q factor value for eachindividual filter. The Q factor is related to the breadth of frequenciesmodified by the filter. A narrow filter bandwidth corresponds to asmaller Q factor value and means the gain is being applied to a smallerrange of frequencies. A broader filter bandwidth corresponds to a largerQ value.

Once a frequency band is selected, the bandwidth of the frequency bandcan be widened or narrowed with the Q factor slider. Generally, theamplitude is inversely related to the Q Factor. In a gaming context, theQ factor is generally useful for three types of filters, including apeak filter, a notch filter and a band pass filter. For these filters,the higher the value of Q factor, the narrower the curve or thebandwidth affected by the filter. A peak filter with a low Q factor willbe useful to control the timber of the sound, for example, if the userwants to add bass to give a more powerful sound or feel to a car engine.A peak filter with a high Q factor will be useful to highlight orde-emphasize a certain frequency, for example, if the user wants tohighlight a bomb sound in a particular game such as CS:Go.

As noted, the Q factor is a parameter that relates the filter's centerfrequency to its bandwidth. FIG. 7N-2 illustrates adjustment of a Qfactor for a filter with a relatively narrow bandwidth. The Q factor hasa value of 10.00. FIG. 7N-3 illustrates adjustment of a Q factor for afilter with a relatively wide bandwidth. The Q factor has a value of0.5. FIG. 7N-2 and FIG. 7N-3 also illustrate an example in which, as amouse or other accessory is used to select a filter for adjustment,current values of gain, frequency and Q factor for the selected filterare displayed on the screen near the cursor to simplify adjustment bythe user and to increase precision of adjustments by the user.

The filter type selector 7638 allows the user to select among differentavailable filter types for use in the parametric equalizer. Onceselected, for each band, the filter type can be chosen in the in afilter type dropdown menu. In the example, filter types availableinclude bypass, lowpass, high pass, peak, low shelving, high shelvingfilters. Other filter types may be specified including combinations offilters. In a gaming context, a peak filter with a low Q factor, a lowshelf filter, or a high shelf filter may be useful to enhance theentertainment feeling for the user. Further, peak filter with a high Qfactor and a notch filter may be useful to give a competitive advantageto a gamer in a gameplay situation. In many applications, a band passfilter is not useful during a game session. However, a band pass filtercan be very useful to search for the frequency of a specific sound. Forexample, a user may want to know what frequency is associated withfootsteps in the game audio stream. The band pass filter allows the userto listen to only a small frequency region or band of frequencies.

Once the parametric equalizer has been used to establish a set ofpreferences for audio processing for the gaming audio stream, the chataudio stream and the microphone audio stream, the established values maybe stored by the user for subsequent use. The user may store data for asingle audio stream, or the user may store data for multiple audiostreams. The data may be stored on the user's computer or may becommunicated over a network for storage at a remote location such as inthe cloud. The data may be retrieved for editing or other modification.In some embodiments, the user may share a profile or configuration withanother gamer. This may be done in any convenient manner. Further, theprofile or the configuration may be specific to a particular game thatthe user plays among multiple games. Thus, the user may store a firstconfiguration or profile for the Fortnite game and store a secondconfiguration or profile for the World of Warcraft game.

FIG. 7O illustrates an exemplary embodiment of a graphical userinterface in which the user may apply a predetermined configuration orprofile to a respective audio stream. The graphical user interfaceincludes a configuration selector 7640. The configuration selector 7640includes an audio stream selector 7641 and a configuration pop-up menu7642. The user may use the mouse or another accessory to select an audiostream of interest for applying a profile or configuration. In theexample embodiment, the user may select the mixer, the gaming audiostream, the chat audio stream or the microphone audio stream forapplication of a configuration. Once the audio stream of interest isselected, the user may use a pop-up menu to select a predeterminedprofile or configuration for the selected audio stream. In the example,for the gaming audio stream, the user has selected a configurationtitled “Fortnite footsteps” which may include data defining a set ofaudio settings for the parametric equalizer that improve the clarity offootsteps heard by the user in the game play. Adjusting the frequency,gain and Q factor of a set of filters may greatly improve the user'sability to discern footsteps during gameplay.

Thus, using the graphical user interface, the user may create,duplicate, edit and delete configurations for each unique audio stream.This allows the user to change audio stream effects with just one touchto select a predetermined configuration, without having to again selecteach specific filter setting. In embodiments, the configurations can beset up and applied to each separate audio stream to maximize thepersonalization for each gamer.

In some embodiments, the audio processing system will automaticallyretrieve and implement a user's configurations when a game is launchedby the user. For example, the user provides identification informationsuch as logging in to a game application or web site. Based on theidentification information or other information the user provides, theaudio processing software may select the user's pre-storedconfigurations for the game. In some embodiments, the process ofretrieving the user's pre-stored configurations may be doneautomatically based on user information or other information. In otherembodiments, the user may be prompted to select a set of configurationsor a profile. In still other embodiments, the user may have to activelylocate and apply the user's pre-stored set of configurations for a gameor the user's profile for the game.

In other examples, a group of pre-set configurations may be madeavailable to users for some games, or to some users of some games. Forexample, the manufacturer may design a set of profiles or configurationfiles that a user may access and select for application to the user'saudio processing system. Access to the pre-set configuration files maybe on a subscription basis for example. Further, gamers may prepare andstore in a common location a library of profiles or configurations fordifferent games and different preferences. The stored profiles may beretrieved, modified and improved by members of a gamer community toenhance the enjoyment of the group.

FIG. 7P illustrates an exemplary embodiment of a graphical userinterface in which the user may apply basic audio rendering presets forquick and easy audio effects applied to each individual audio stream.The graphical user interface includes a gaming audio render effectsselector 7645. The gaming audio render effects selector 7645 in theexample includes an audio stream selector 7646 and an audio adjustmentpanel 7647. Other examples may include other adjustments or features.

The audio stream selector 7646 allows the user to select an audio streamfor application of the audio rendering presets. In the example, the usermay select between the mixer, the gaming audio stream, the chat audiostream and the microphone audio stream.

The audio adjustment panel 7647 includes a set of sliders that may beactuated by a user with a mouse or other accessory. Each of the slidersis associated with an audio adjustment. Each of the sliders is alsoassociated with an on-off selector that may be actuated by the user onthe graphical user interface to enable or disable particular audioadjustments. In the example embodiment, the available audio adjustmentsinclude bass boost, treble boost, voice clarity, smart volume, generalgain and virtual surround.

Tuning the bass boost audio adjustment allows the user to boost or cutlow frequencies. In an example, the bass frequencies are around 80 Hz.This may have the effect of making more or less audible deep, lowfrequency sounds in the gaming audio stream such as explosions andengines. The bass boost audio adjustment can be tuned through the sliderin a range from +12 dB to −12 dB in the example.

Tuning the treble boost audio adjustment allows the user to boost or cuthigh frequencies. In an example, the treble frequencies are around 8000Hz. This may have the effect of making more or less audible highfrequency sounds in the gaming audio stream like footsteps, gunshots orwhistles. For example, this can give the user a real advantage inshooting games. The treble boost can be tuned through the slider in arange from −12 dB up to +12 dB in the example.

The voice clarity audio adjustment identifies and treats the dialogs orvoices in the audio stream produced by the game. That means that tuningthe voice clarity through the slider enables the user to boost or cutthe speech in games, movies and incoming communications. The voice boostaudio adjustment can be tuned through the slider in a range from −12 dBup to +12 dB in the example.

Some audio streams have very large dynamic range, or the differencebetween the loudest sounds and the softest sounds. The smart volumeaudio adjustment automatically makes small adjustments and helpsmaintain a steady volume in the fame for the user. For example, if theuser is constantly adjusting the volume on the user's computer whileplaying games or listening to music, the smart volume will solve addressthe problem. The user may tune the impact of this audio effect on yoursound through the slider in a range from 0 to 1 in the example.

The general gain audio adjustment simply allows the user to adjustoverall sound volume. Such a feature is useful when the audio sourcevolume is low. The general gain audio adjustment can be tuned throughthe slider through a range from 0 dB up to +12 dB in the example. Also,the general gain audio adjustment does not allow system limits on audiovolume or gain to be exceeded.

The virtual surround sound audio adjustment may be used in arrangementsthat include a stereo device such as a headset, or notebook computerinternal speakers. In one embodiment, ff a multichannel audio streamsuch as 5.1 channel audio or 7.1 channel audio is available, switchingON the virtual surround audio adjustment re-creates a multichannelthree-dimensional audio listening experience over the user's stereoheadphones or speakers. When actuated ON, the virtual surround soundaudio adjustment operates to virtualize the multichannel game audiostream from the game engine, or any other media content, and mix theaudio stream to retrieve a multichannel listening experience over theuser's stereo headphones or speakers. On the other hand, if amultichannel audio stream, such as stereo content, is not available,switching ON the virtual surround sound audio adjustment operates toexpand the stereo sound for a wider sound stage.

In another example, if the user's audio device is a multichannel devicesuch as a home theater system, and if a multichannel audio stream suchas 5.1 channel or 7.1 channel audio is available, switching ON thevirtual surround sound audio adjustment will virtualize the multichannelaudio stream on left and right speakers. In embodiments, only left andright speakers will output sound. In this situation, it may bepreferable not to activate the virtual surround sound audio adjustmentsince a real surround sound effect is available. In this case, if amultichannel audio stream is not available, such as stereo content,switching ON the virtual surround sound audio adjustment operates toexpand the stereo sound for a wider sound stage. Only left and rightspeakers may output sound in this example.

FIG. 7Q illustrates an exemplary embodiment of a graphical userinterface including a chat equalizer 7650. The chat equalizer 7650 inthe example is embodied as a ten-band graphic equalizer for the teamchat audio stream. The chat equalizer 7650 includes an audio streamselector 7651, a series of audio stream controllers 7652. Otherembodiments may include other features in addition to those illustratedin FIG. 7Q.

The audio stream selector 7651 allows the user to select an audio streamfor application of audio equalization. In the example, the user mayselect between the mixer, the gaming audio stream, the chat audio streamand the microphone audio stream. The chat audio stream is selected inFIG. 7Q. The chat equalizer allows the user to tailor the audio of theteam chat to improve the way in which the user hears teammates or othersspeaking on the chat audio stream. In a typical embodiment, a game isplayed on a gaming system with two or more players working together asteammates. Each player has access to a microphone and speakers, such asan integrated headset wearable by the player. The players may use themicrophone and speakers to converse with each other or more generally toshare speech and non-speech voiced audio. The chat audio stream mayinclude non-speech such as screams or laughter. The audio streamcontroller 7652 allows the user to adjust gain upward and downwardacross ten frequency bands.

In the illustrated embodiment, the chat equalizer 7650 includes aconfiguration selector 7653. The user may create and store one or moreconfigurations or profiles. For example, after the user adjusts theaudio stream controller 7652 to tailor sound processing for the chataudio stream according to the user's preferences, the user may save thedata that device the current adjustments. The data may be saved as aprofile file or configuration file and may be saved locally or remotely.A saved profile may be shared among users or may be part of a library ofavailable configurations. A profile may be retrieved and edited by theuser. The configuration selector 7653 in the example includes a pop-upmenu that, when actuated by the user, displays available profiles orconfigurations. In some examples, a profile or configuration may bedesignated as a default profile and load automatically when the game isstarted. A profile may be established for a respective game according touser preferences.

FIG. 7R illustrates an exemplary embodiment of a graphical userinterface in which the user may apply basic audio rendering presets forquick and easy audio effects applied to the chat audio stream. Thegraphical user interface includes a chat audio render effects selector7655. The chat audio render effects selector 7655 in the exampleincludes an audio stream selector 7656 and an audio adjustment panel7657. Other examples may include other adjustments or features.

The audio stream selector 7656 allows the user to select an audio streamfor application of the audio rendering presets. In the example, the usermay select between the mixer, the gaming audio stream, the chat audiostream and the microphone audio stream. In the illustrated example, thechat audio stream is selected.

The audio adjustment panel 7657 allows the user to select and adjust avariety of aspects of the chat audio stream. In the illustrated example,the user has access to a noise reduction audio adjustment, a volumestabilizer audio adjustment, a noise gate audio adjustment, an automaticnoise gate audio adjustment and an impact noise reduction audioadjustment.

The noise reduction audio adjustment operates to remove static noisefrom the user's sound by cleaning up spoken words and sentences. Thisaudio adjustment is particularly useful to remove constant noise likepersonal computer fans. The user may tune the effect of the noisereduction audio adjustment through the slider from 0, appropriate for aquiet environment, to 1, appropriate for a noisy environment.

The smart volume audio adjustment automatically makes small adjustmentsand helps maintain a steady volume, particularly when the chat audiostream has a relatively high dynamic range. The smart volume audioadjustment eliminates the need for the user to constantly adjust acomputer's volume during gameplay or listening to music. The user maytune the impact of the smart volume audio adjustment on user soundthrough the slider, over a range from 0 to 1.

For a given a volume threshold, the noise gate audio adjustment operatesto cut off any sound behind it. If sound is above a predetermined noisegate threshold, the audio processing system lets pass all the sound. Inother words, the sound is allowed through the gate. On the other hand,if the sound is below the predetermined noise gate threshold, the sounddoes not pass through the gate. The predetermined noise gate thresholdcan be changed with the threshold slider in the range −60 dB-10 dB. Thiseffect can be particularly useful if the user perceives an annoying ordistracting noise which is way behind the conversation level.

For a given volume threshold, the noise gate audio adjustment will cutoff any sound behind it. The auto noise gate audio adjustmentautomatically computes the predetermined noise gate threshold accordingto detected background noise. In embodiments, the auto noise gate audioadjustment requires the noise gate audio adjustment to be ON, i.e., bothaudio adjustments must be activated. The auto noise gate audioadjustment operates automatically. However, the user has a layer ofcontrol over the auto noise gate audio adjustment with the offset sliderwhich allows the user to apply a gain over a range from −20 dB to +20dB) on the top of the computed threshold. The auto noise gate is usefulif the user needs a noise gate, but the user is uncertain about whichthreshold to use.

The impact noise reduction audio adjustment operates to detect andsuppress impact noises from the user's chat audio stream or inputdevice. The impact noise reduction audio adjustment is useful inreducing the noise coming from sources such as keystroke on a keyboardor noise from mouse clicks.

FIG. 7S illustrates an exemplary embodiment of a graphical userinterface including a microphone equalizer 7660. The microphoneequalizer 7660 in the example is embodied as a ten-band graphicequalizer for equalizing the user's microphone audio stream. Themicrophone equalizer 7660 includes an audio stream selector 7661 and aseries of audio stream controllers 7662. Other embodiments may includeother features in addition to those illustrated in FIG. 7S.

The audio stream selector 7661 allows the user to select an audio streamfor application of audio equalization. In the example, the user mayselect between the mixer, the gaming audio stream, the chat audio streamand the microphone audio stream. The microphone audio stream is selectedin FIG. 7S. The microphone equalizer allows the user to tailor the audioof the user's microphone provided to the team chat to improve the way inwhich teammates or others hear the user speaking on the chat audiostream. In a typical embodiment, a game is played with two or moreplayers working together as teammates. Each player has access to amicrophone and speakers. The players may use the microphone and speakersto converse with each other in the chat audio which may includenon-speech such as screams or laughter. The audio stream controller 7662allows the user to adjust gain upward and downward across ten frequencybands.

In the illustrated embodiment, the chat equalizer 7650 includes aconfiguration selector 7663. Similar to profiles for the game audiostream and chat audio stream, the user may create and store one or moreconfigurations or profiles. For example, after the user adjusts theaudio stream controller 7662 to tailor sound processing for themicrophone audio stream according to the user's preferences, the usermay save the data that device the current adjustments. The data may besaved as a profile file or configuration file and may be saved locallyor remotely. A saved profile may be shared among users or may be part ofa library of available configurations. A profile may be retrieved andedited by the user. The configuration selector 7663 in the exampleincludes a pop-up menu that, when actuated by the user, displaysavailable profiles or configurations. In some examples, a profile orconfiguration may be designated as a default profile and loadautomatically when the game is started. A profile may be established fora respective game according to user preferences.

FIG. 7T illustrates an exemplary embodiment of a graphical userinterface in which the user may apply basic audio rendering presets forquick and easy audio effects applied to the microphone audio streamgenerated by the user's own microphone. The microphone audio stream maybe shared with teammates, for example, during game play or at othertimes. In the illustrated example, the graphical user interface includesa microphone audio render effects selector 7655. The chat audio rendereffects selector 7665 in the example includes an audio stream selector7666 and an audio adjustment panel 7667. The microphone audio rendereffects selector 7665 in the exemplary embodiment operates similarly toand includes features like those of chat audio render effects selector7655 illustrated in FIG. 7R. Other examples may include otheradjustments or features.

The audio stream selector 7656 allows the user to select an audio streamamong the audio streams processed by the audio processing system forapplication of the audio rendering presets. In the example, the user mayselect between the mixer, the gaming audio stream, the chat audio streamand the microphone audio stream. In the illustrated example, themicrophone audio stream is selected.

The audio adjustment panel 7667 allows the user to select and adjust avariety of aspects of the microphone audio stream. In the illustratedexample, the user has access to a noise reduction audio adjustment, avolume stabilizer audio adjustment, a noise gate audio adjustment, anautomatic noise gate audio adjustment and an impact noise reductionaudio adjustment.

The noise reduction audio adjustment operates to remove static noisefrom the audio stream created by the user's microphone by filtering orcleaning up spoken words and sentences. This audio adjustment isparticularly useful to remove constant noise like personal computer fansor other persistent noises in the space with the user that might bepicked up by the user's microphone. The user may tune the effect of thenoise reduction audio adjustment through the slider from 0, appropriatefor a quiet environment, to 1, appropriate for a noisy environment.

The smart volume audio adjustment automatically makes small adjustmentsand helps maintain a steady volume, particularly when the microphoneaudio stream has a relatively high dynamic range. The smart volume audioadjustment eliminates the need for the user to constantly adjust acomputer's volume during gameplay or while listening to music. The usermay tune the impact of the smart volume audio adjustment on microphoneaudio through the slider, over a range from 0 to 1.

For a given a volume threshold, the noise gate audio adjustment operatesto cut off any sound behind it. If sound is above a predetermined noisegate threshold, the audio processing system lets all the sound pass. Inother words, the sound is allowed through the gate. On the other hand,if the sound is below the predetermined noise gate threshold, the sounddoes not pass through the gate. The predetermined noise gate thresholdcan be changed with the threshold slider in the range −60 dB-10 dB. Thiseffect can be particularly useful if the user perceives an annoying ordistracting noise which is way behind conversation level.

For a given volume threshold, the noise gate audio adjustment will cutoff any sound behind it. The auto noise gate audio adjustmentautomatically computes the predetermined noise gate threshold accordingto detected background noise. In embodiments, the auto noise gate audioadjustment requires the noise gate audio adjustment to be ON, i.e., bothaudio adjustments must be activated. The auto noise gate audioadjustment operates automatically. However, the user has a layer ofcontrol over the auto noise gate audio adjustment with the offset sliderwhich allows the user to apply a gain over a range from −20 dB to +20dB) on the top of the computed threshold. The auto noise gate is usefulif the user needs a noise gate, but the user is uncertain about whichthreshold to use.

The impact noise reduction audio adjustment operates to detect andsuppress impact noises from the user's chat audio stream or inputdevice. The impact noise reduction audio adjustment is useful inreducing the noise coming from sources such as keystroke on a keyboardor noise from mouse clicks.

FIGS. 8-9 illustrate embodiments of a system with a correspondingcommunication flow diagram for correlating stimulations and gamingaction results. In this illustration a user clicks the left button 119of the gaming controller 115. The gaming controller 115 can includefirmware (or circuitry), which creates an event as depicted by event 2in FIG. 8 . The button depression and the event creation are depicted inFIG. 9 as steps 902 and 904. In step 904, the firmware of the gamingcontroller 115 can, for example, generate an event type “left button#3”, and a unique GUID with a time stamp which is submitted to the AMSapplication. Referring back to FIG. 8 , the AMS application cataloguesevent 3, and if a substitute stimulation has been predefined, remaps theevent according to the substitution. The remapped event is thentransmitted to the gaming application at event 4. Event 3 of FIG. 8 isdepicted as step 906 in FIG. 9 . In this illustration, the AMSapplication substitutes the left button #3 depression stimulus with a“keyboard ‘F’” depression which can be interpreted by the gamingapplication as a fire command. The AMS application in this illustrationcontinues to use the same GUID but substitutes the time stamp foranother time stamp to identify when the substitution took place.

Referring back to event 4, the gaming application processes the eventand sends back at event 5 a game action result to the AMS applicationwhich is processed by the AMS application at event 6. The AMSapplication then submits the results to the accessory at event 7. Events4 and 5 are depicted as step 908 in FIG. 9 . In this step, the gamingapplication processes “F” as an action to fire the gamer's gun, and thendetermines from the action the result from logistical gaming resultsgenerated by the gaming application. In the present illustration, theaction of firing resulted in a hit. The gaming application submits tothe AMS application the result type “Hit” with a new time stamp, whileutilizing the same GUID for tracking purposes. At step 910, the AMSapplication correlates the stimulation “left button #3 (and/or thesubstitute stimulation keyboard “F”) to the game result “Hit” andcatalogues them in memory. The AMS application then submits to theaccessory (e.g., gaming controller 115) in step 910 the game actionresults “Hit” with the same GUID, and a new time stamp indicating whenthe result was received. Upon receiving the message from the AMSapplication, the accessory in step 912 processes the “Hit” by assertinga red LED on the accessory (e.g., left button 119 illuminates in red orother LED of the gaming controller 115 illuminates in red) to indicate ahit. Other notification notices can be used such as another color forthe LED to indicate misses, a specific sound for a hit, or kill, avibration or other suitable technique for notifying the gamer of thegame action result.

Upon reviewing the aforementioned embodiments, it would be evident to anartisan with ordinary skill in the art that the embodiments of thesubject disclosure can be modified, reduced, or enhanced withoutdeparting from the scope of the claims described below. For example, theAMS application can be executed from an accessory 115 or computingdevice 206 to perform the embodiments described in the subjectdisclosure. The AMS application can also be operated from a remoteserver (“cloud services”). In yet another embodiment, functions of theAMS application can be distributed between devices. In yet anotherembodiment, the AMS application can be configured to track theperformance of a gamer and adapt a threshold as the gamer improves ordeclines in performance.

For instance, as a gamer's performance improves with a particular gamingaction, the threshold associated with the gaming action can be adaptedto be less sensitive in detecting an over usage state. Similarly, thesensitivity of the threshold can be increased to promptly identify anover usage state of a gaming action if the gamer's performance declinesas a result of an over usage of the gaming action. Additionally, the AMSapplication can be adapted to add gaming actions to an exclusion tablewhen the gamer's performance substantially improves as a result of usingthe gaming action being excluded. The exclusion table can also bechanged by the AMS application by removing a gaming action from theexclusion table responsive to its excessive use causing a decline in agamer's performance.

Other embodiments can be applied to the subject disclosure.

It should be understood that devices described in the exemplaryembodiments can be in communication with each other via various wirelessand/or wired methodologies. The methodologies can be links that aredescribed as coupled, connected and so forth, which can includeunidirectional and/or bidirectional communication over wireless pathsand/or wired paths that utilize one or more of various protocols ormethodologies, where the coupling and/or connection can be direct (e.g.,no intervening processing device) and/or indirect (e.g., an intermediaryprocessing device such as a router).

FIG. 10 depicts an exemplary diagrammatic representation of a machine inthe form of a computer system 1000 within which a set of instructions,when executed, may cause the machine to perform any one or more of themethods described above. One or more instances of the machine canoperate, for example, as an accessory, computing device or combinationsthereof. In some embodiments, the machine may be connected (e.g., usinga network 1026) to other machines. In a networked deployment, themachine may operate in the capacity of a server or a client user machinein a server-client user network environment, or as a peer machine in apeer-to-peer (or distributed) network environment.

The machine may comprise a server computer, a client user computer, apersonal computer (PC), a tablet, a smart phone, a laptop computer, adesktop computer, a control system, a network router, switch or bridge,or any machine capable of executing a set of instructions (sequential orotherwise) that specify actions to be taken by that machine. It will beunderstood that a communication device of the subject disclosureincludes broadly any electronic device that provides voice, video ordata communication. Further, while a single machine is illustrated, theterm “machine” shall also be taken to include any collection of machinesthat individually or jointly execute a set (or multiple sets) ofinstructions to perform any one or more of the methods discussed herein.

The computer system 1000 may include a processor (or controller) 1002(e.g., a central processing unit (CPU)), a graphics processing unit(GPU, or both), a main memory 1004 and a static memory 1006, whichcommunicate with each other via a bus 1008. The computer system 1000 mayfurther include a display unit 1010 (e.g., a liquid crystal display(LCD), a flat panel, or a solid-state display). The computer system 1000may include an input device 1012 (e.g., a keyboard), a cursor controldevice 1014 (e.g., a mouse), a disk drive unit 1016, a signal generationdevice 1018 (e.g., a speaker or remote control) and a network interfacedevice 1020. In distributed environments, the embodiments described inthe subject disclosure can be adapted to utilize multiple display units1010 controlled by two or more computer systems 1000. In thisconfiguration, presentations described by the subject disclosure may inpart be shown in a first of the display units 1010, while the remainingportion is presented in a second of the display units 1010.

The disk drive unit 1016 may include a tangible computer-readablestorage medium 1022 on which is stored one or more sets of instructions(e.g., software 1024) embodying any one or more of the methods orfunctions described herein, including those methods illustrated above.The instructions 1024 may also reside, completely or at least partially,within the main memory 1004, the static memory 1006, and/or within theprocessor 1002 during execution thereof by the computer system 1000. Themain memory 1004 and the processor 1002 also may constitute tangiblecomputer-readable storage media.

Dedicated hardware implementations including, but not limited to,application specific integrated circuits, programmable logic arrays andother hardware devices can likewise be constructed to implement themethods described herein. Application specific integrated circuits andprogrammable logic array can use downloadable instructions for executingstate machines and/or circuit configurations to implement embodiments ofthe subject disclosure. Applications that may include the apparatus andsystems of various embodiments broadly include a variety of electronicand computer systems. Some embodiments implement functions in two ormore specific interconnected hardware modules or devices with relatedcontrol and data signals communicated between and through the modules,or as portions of an application-specific integrated circuit. Thus, theexample system is applicable to software, firmware, and hardwareimplementations.

In accordance with various embodiments of the subject disclosure, theoperations or methods described herein are intended for operation assoftware programs or instructions running on or executed by a computerprocessor or other computing device, and which may include other formsof instructions manifested as a state machine implemented with logiccomponents in an application specific integrated circuit or fieldprogrammable gate array. Furthermore, software implementations (e.g.,software programs, instructions, etc.) including, but not limited to,distributed processing or component/object distributed processing,parallel processing, or virtual machine processing can also beconstructed to implement the methods described herein. It is furthernoted that a computing device such as a processor, a controller, a statemachine or other suitable device for executing instructions to performoperations or methods may perform such operations directly or indirectlyby way of one or more intermediate devices directed by the computingdevice.

While the tangible computer-readable storage medium 1022 is shown in anexample embodiment to be a single medium, the term “tangiblecomputer-readable storage medium” should be taken to include a singlemedium or multiple media (e.g., a centralized or distributed database,and/or associated caches and servers) that store the one or more sets ofinstructions. The term “tangible computer-readable storage medium” shallalso be taken to include any non-transitory medium that is capable ofstoring or encoding a set of instructions for execution by the machineand that cause the machine to perform any one or more of the methods ofthe subject disclosure. The term “non-transitory” as in a non-transitorycomputer-readable storage includes without limitation memories, drives,devices and anything tangible but not a signal per se.

The term “tangible computer-readable storage medium” shall accordinglybe taken to include, but not be limited to solid-state memories such asa memory card or other package that houses one or more read-only(non-volatile) memories, random access memories, or other re-writable(volatile) memories, a magneto-optical or optical medium such as a diskor tape, or other tangible media which can be used to store information.Accordingly, the disclosure is considered to include any one or more ofa tangible computer-readable storage medium, as listed herein andincluding art-recognized equivalents and successor media, in which thesoftware implementations herein are stored.

Although the present specification describes components and functionsimplemented in the embodiments with reference to particular standardsand protocols, the disclosure is not limited to such standards andprotocols. Each of the standards for Internet and other packet switchednetwork transmission (e.g., TCP/IP, UDP/IP, HTML, HTTP) representexamples of the state of the art. Such standards are from time-to-timesuperseded by faster or more efficient equivalents having essentiallythe same functions. Wireless standards for device detection (e.g.,RFID), short-range communications (e.g., Bluetooth®, Wi-Fi, Zigbee®),and long-range communications (e.g., WiMAX, GSM, CDMA, LTE) can be usedby computer system 1000.

FIG. 13 depicts system 1300 to identify a group of possible triggerclips in accordance with aspects of this disclosure and FIG. 14 depictsa system 1400 to distribute a group of trigger clips to a group ofplayers in accordance with aspects of this disclosure. Referring to bothFIGS. 13 and 14 , in one or more embodiments, a computer server or videogame server 1402 can implement a video game (hereafter video game server1402) involving several different players 1304, 1306, 1308, 1314, 1316,and 1318 in different locations communicating with each other and thevideo game server 1402 over a communication network (e.g. Internet).Further, some players 1304, 1306, 1314, and 1316 can comprise one teamplaying the video game playing against another team of players 1308 and1318. The gaming devices 1404, 1408, 1412, and 1416 are associated witheach of players 1304, 1306, 1314, and 1316 of one team.

In one or more embodiments, the video game server 1402 can detect acurrent trigger during a video game and record a trigger clip. A triggeris an event in the video game that a player may find significant and/orif reviewed, improve player performance in the future. Examples oftriggers can include when a player makes a kill, makes a hit, takes ahit, loss of life, fired upon, implements weapon fire, loss of life ofanother player, change in ammunition, etc. A player may want to reviewtriggers so as to avoid such actions/events (e.g. loss of life, takes ahit, fired upon, etc.) or to continue the actions/events (e.g. makes akill, makes a hit, etc.) to improve video game performance in thefuture. A trigger clip is a portion of the video game that is recordedthat contains the trigger. As described herein, a video game may have asliding window of game content that it continuously records portions ofthe video game. Once the video game server 1402 identifies a trigger,the video game server 1402 can process the recorded slidingwindow/recorded portion of video game content that contains the triggeras the trigger clip. Further, the trigger clip can be provided andpresented to a player for the player's review.

In one or more embodiments, the trigger clip is a recorded portion ofthe game content from the perspective or point of view of a player. Insome embodiments, a trigger can involve one or more players. Thus, adistributed trigger clip may be requested or provided to all playersinvolved in the trigger. That is, a distributed trigger clip is a groupof trigger clips provided to each player involved in the trigger. Eachtrigger clip provided to each player is a portion of game content fromthe associated player's perspective or point of view. Referring to FIG.13 , a trigger clip 1302 of a war scenario video game can include afirst player 1304 on a first team. The trigger clip 1302 can be from theperspective of the first player 1304. The trigger can be the loss oflife of a second player 1306 of the first team. Further, the triggerclip 1302 can include a first player 1308 on a second team that istaking cover behind a building 1310. The first player 1304 of the firstplayer may request to clip the trigger of the loss of life of the secondplayer 1306 of the first team because the first player 1304 on the firstteam is unsure whether he/she killed the second player 1306 on the firstteam (i.e. loss of life due to friendly fire), or whether the firstplayer 1308 on the second team killed the second player 1306 on thefirst team (i.e. loss of life due to enemy fire). The first player 1304may want to review the trigger clip to review whether he/she committedfriendly fire, and if so, avoid doing so in the future.

In one or more embodiments, the video game server 1402 can identify thatthe first player 1304 and the second 1306 on the first team are bothinvolved in the trigger. Thus, the video game server 1402 can generateanother trigger clip 1302 a from the perspective of the second player1306. Referring to FIG. 14 , when requested by the first player 1304,the video game server can provide the distributed trigger clip, whichcomprises trigger clip 1302 and trigger clip 1302 a. Trigger clip 1302can be sent to gaming device 1404 associated with the first player 1304over a communication network (wired, wireless, or a combination thereof)to be presented on display 1406 for review by the first player 1304.Further, trigger clip 1302 a can be sent to gaming device 1408associated with the second player 1306 (wired, wireless, or acombination thereof) to be presented on display 1410 for review by thefirst player 1306.

Referring to FIG. 13 , a third player 1314 and a fourth player 1316 onthe first team (same team as first player 1304 and second player 1306)are not involved in the trigger because they are in a differentscene/setting 1312 of the video game battling a second player 1318 onthe second team in a forest. Referring to FIG. 14 , the video gameserver 1402 would identify that neither the third player 1314 nor thefourth player 1316 would not be provided with the distributed triggerclip because they were not involved with the trigger. Thus, even thoughthe video game server 1402 is communicatively coupled to gaming devices1412, 1416 and displays 1414 and 1418 associated with players 1314 and1316 over a communication network (wired, wireless, or a combinationthereof), the video game server 1402 would not provide the distributedtrigger clip 1302, 1302 a to gaming devices 1412 and 1416.

In one or more embodiments, the video game server 1402 can detect oridentify a trigger by receiving user-generated input from the firstplayer 1304, the user-generated input indicating to generate a clipaccording to the event/trigger and the players in involved in theevent/trigger. The user-generated input can also include a timestamp.The user-generated input can be provided entering text input, gestureinput (using a touchscreen graphical user interface), and/or voice input(involving voice recognition techniques) into the gaming device 1404.Further, the user-generated input provided to the gaming device 1404 isthen transmitted to the video game server 1402 for further processing.The video game server 1402 can then record the distributed trigger clip(group of trigger clips) about the timestamp (e.g. 20 seconds before andafter the indicated or detected trigger) included in the user-generatedinput (or detected otherwise) for the trigger/event and for the playersindicated in the user-generated input.

In one or more embodiments, prior to detecting a trigger, the video gameserver 1402 can receive user-generated input (as described herein)indicating different events that are associated with different triggertypes. For example, trigger types can include of kill, hit, weapon fire,loss of life of current player, loss of life of another player, andchange in ammunition. The video game server 1402 can associate eachevent with a trigger type according to the user-generated input.

In one or more embodiments, the video game server 1402 can detect atrigger by implementing image recognition techniques to identify orotherwise determine that an event within a portion of game content is atrigger. In other embodiments, the video game server 1402 can accesshistorical game content and identify a previous trigger within thehistorical game content. That is, the video game server 1402 can analyzehistorical game content to determine when an event was identified as atrigger according to its trigger type. Based on such a determination,the video game server 1402 can detect a current trigger by determiningthe current trigger/event is of the same type as the previous trigger.

In one or more embodiments, the detecting of a current trigger caninclude accessing training game content, acquiring machine learningtechniques to identify different trigger from the different event fromthe training game content. That is, different events that are identifiedas different triggers in the training game content can be used to trainan artificial intelligence engine on the video game server 1402 toacquire machine learning techniques to identify triggers in portions ofgame content in the future. Further, the detecting of a current triggercan comprise identifying the current trigger by using, or in responseto, implementing the machine learning techniques on the portion of gamecontent.

In one or more embodiments, the detecting of a trigger can includedetecting a particular sound or utterance or word or statement by aplayer to identify or otherwise determine that an event within a portionof game content is a trigger. This may be done in any suitable mannerincluding using artificial intelligence or machine learning techniques.In an example, a neural network is trained to estimate, on the fly, ifthere is, for example, a laugh or scream from a player. In an example,operating “on the fly” may mean within a time delay of less than onesecond, or similar time delay. Operation is fast but not instantaneousas the device needs to accumulate data which introduces a certainlatency to the process. Any suitable neural network may be used.Training data may be audio data from other gameplay that is labelled toidentify a laugh or a scream or other utterance or statement that is tobe designated as a trigger. The neural network is trained to provide asan output an indication that the trigger has been detected oridentified. After training, a portion of an audio stream is provided tothe neural network and the neural network generates a value thatcorresponds to a probability that the trigger has been detected. Aprocessing system may compare the value with a predetermined threshold.If the probability exceeds the predetermined threshold, such as 50percent or 85 percent or 95 percent, the characteristic may beidentified as a significant trigger. The training data may include anysort of audio including conversations, laughter, screaming and othersounds as well, including sounds that are representative of typicalsounds.

A system including an audio processing system such as audio processingsystem 7600 (FIG. 7L) may include a chat audio stream and a microphoneaudio stream. The chat audio stream may include audio corresponding toconversation and other utterances from gamers such as teammates of alocal user. The microphone audio stream may include audio correspondingto conversation and other utterances from the local user, detected bythe microphone adjacent to the local user. In a first use case, a neuralnetwork, or multiple neural networks, may be trained to detect anoccurrence in the chat audio stream or an occurrence in the microphoneaudio stream and, responsive to the detected occurrence, to cause asubsequent action within the gaming system. In other embodiments, in asecond use case, the neural network may operate on a combined audiostream that includes both the chat audio stream and the microphone audiostream. Similarly, in a third use case, a single audio stream includes acombined audio stream that includes the chat audio stream, themicrophone audio stream and a game audio stream that includes soundsgenerated by the game. The training data for either of these use casesis tailored to the particular use case.

In a particular embodiment, a trigger is detected by automaticallydetecting a laugh or a scream in an audio stream. The laugh or screamcan originate with any player. A motivation for such automatictriggering is the notion that, if someone is laughing or screaming,something interesting happened in the game and that should be recordedin a trigger clip. Both audio and video are recorded in the triggerclip.

In some embodiments, a first window of video of a first predeterminedtime duration is recorded before the occurrence of the trigger and asecond window of a second predetermined time duration, that may bedifferent from the first predetermined time duration, is recordedfollowing the occurrence of the trigger. The nature of the detectedtrigger, such as a laugh or a scream, can be used to select the durationof the trigger clip. For example, a laugh detected as a trigger maycause a 30 second trigger clip having a duration of 30 seconds to berecorded and a scream detected as a trigger may cause a trigger cliphaving a duration of 60 seconds to be recorded. In some embodiments, anyaspect of detecting a trigger, identifying a trigger or the duration ofthe trigger clip associated with a specific trigger may be userselectable, such as through a graphical user interface such as GUI 7610(FIG. 7L).

In some embodiments, one or more players has a webcam or other devicecollecting video of the player or the player's environment. The videostream from the webcam can be provided to a neural network to identify avideo trigger in a manner similar to the identifying an audio trigger inan audio stream. The neural network may be trained with appropriate datathat is labelled to identify the trigger of interest. In furtherembodiments, both an audio stream and a video stream may be monitored ina multimodal approach to identify a trigger event. For example, a videoneural network will produce a video probability that a video trigger hasbeen detected and an audio neural network will produce an audioprobability that an audio trigger has been detected. The videoprobability and the audio probability may be combined and evaluated inany suitable manner against one or more probability thresholds. Forexample, if a player laughs silently, the audio neural network may notdetect the sound corresponding to a laugh, but the video neural networkmay detect the action or movement associated with laughter, or aprobability that the motion corresponds to a laugh. Any change ofemotion or other response captured in audio or video, may be designatedas a trigger.

In some embodiments, a trigger may be defined according to particularcultural aspects of a player. For example, research may show thatEuropean players react to a game situation in a first way, Americanplayers react to the same game situation in a second way, south Asianplayers react to the same game situation in a third way and east Asianplayers react to the same game situation in a fourth way. Reactions ofthe different cultures may include utterances and statements made orallyor body movements or postures, or any combination of these. Differentmodels may be directed to geography or language or other characteristicsof different gamers.

In some embodiments, the processing of audio and video streams toidentify triggers may occur on a user device such as computing device206 (FIG. 2 ). The processing may be part of the game applicationrunning on a gamer's device. In other embodiments, the processing ofaudio and video streams may be done remotely such as at a server hostingthe game and accessible over a network such as the public internet.

In some embodiments, the audio stream may be provided to a speechrecognition module. The speech recognition module may detect andidentify words stated by a gamer. Any suitable speech recognition modulemay be used for this purpose. In this manner, the user may be able toprovide commands to the gaming system. Moreover, a trigger may bedetected based on an identified statement or command of the player.

In some embodiments, data for a user's audio and video recognition maybe stored in a user profile. The user profile may contain any suitableinformation useful for processing the audio or the video associated withthe user. The user profile may be accessed when the user initiates thegame or during gameplay, or at any appropriate time. The user profilemay be stored locally at a device of the user. In other embodiments, theuser profile may be stored remotely at a server or other storagelocation for access by the local device of the user.

In some embodiments, a detected trigger may be characterized todetermine particular characteristics of the trigger. The detectedcharacteristics may be used to automatically select differentcharacteristics of the trigger clip that is recorded. For example, asound may be identified as a scream. If the volume of the scream exceedsa certain threshold, or if the duration of the scream exceeds a certainduration, or a combination of these characteristics, the scream may betreated as a more significant trigger. Identification of the moresignificant trigger may cause the audio processing system or otheraspect to record a trigger clip having a particular characteristic, suchas a longer duration. Similarly, identification of the more significanttrigger may cause the audio processing system or other aspect todistribute the recorded trigger clip to a different audience than theaudience for a conventional trigger. The neural network may be trainedusing appropriate training data to distinguish any characteristics ofinterest in the audio stream and to generate a value corresponding to aprobability that the trigger has been detected. If the probabilityexceeds a predetermined threshold, such as 85 percent, thecharacteristic may be identified as a significant trigger. Any suitablestatistics in the audio stream and in the neural network can be used toidentify one or more triggers.

In one or more embodiments, the video game server can determine a groupof players involved in a current trigger by identifying the group ofplayers using, or in response to, implementing image recognitiontechniques on a trigger clip. In one or more embodiments, the video gameserver can instead, or in addition, identify a group of players involvedin a current trigger by identifying the group of players using, or inresponse to, analyzing one or more audio streams such as chat audiostreams or microphone audio streams on a trigger clip. In someembodiments, the video game server 1402 can determine the group ofplayers involved in the current trigger by identifying the group ofplayers by, or in response to, analyzing game action during the sametime period each possible trigger clip in the group of trigger clips.That is, when the video game server 1402 identifies a trigger in aportion of game content from one player's perspective, the video gameserver can then generate a trigger clip for that player (i.e. record aportion of the game content from that player's perspective). However,the video game server 1402 can also record other portions of gamecontent, each portion of game content from a different player'sperspective. These group of portions of game content are possibletrigger clips to comprise the distributed trigger clip. However, afterdetermining only players involved in the trigger, the video game server1402 discards portions of game content that are from perspective ofplayers not involved in the trigger and generates the distributedtrigger clip to comprise a group of portions of game content (i.e.trigger clips) from the perspective of each player involved in thetrigger.

FIGS. 15A-D depict illustrative embodiments of methods utilized in thesubject disclosure. Further, the methods depicted in FIGS. 15A-D can beimplemented by a video game server as described herein. Referring toFIG. 15A, the method 1500 can include the video game server, at 1502,associating each event in a group of events with each trigger type froma group of trigger types. Further, the method 1500 can include the videogame server, at 1504, detecting a current trigger during a video game.In addition, the method 1500 can include the video game server, at 1506,determining a group of players involved in the current trigger. Also,the method 1500 can include the video game server, at 1508, recording atrigger clip of game content for each player in the group of playersresulting in a group of trigger clips. Each trigger clip of the group oftrigger clips is a portion of game content that includes the currenttrigger from a perspective of each player of the group of players.Further, the method 1500 can include the video game server, at 1510,providing the group of trigger clips to a group of gaming devices over acommunication network. Each gaming device of the group of gaming devicesis associated with one player of the group of players.

Referring to FIG. 15B, the method 1511 can implement detecting of thecurrent trigger in different embodiments. The method 1511 can includethe video game server, at 1512, receiving first user-generated input.The first user-generated input indicates the current trigger and/or torecord the group of trigger clips. Further, the method 1511 can includethe video game server, at 1514, implementing image recognitiontechniques on each portion of game content associated with each oftrigger clip of the group of trigger clips to determine that an eventoccurred within each portion of game content such that the eventcomprises the current trigger. In addition, the method 1511 can includethe video game server, at 1516, accessing historical game content. Also,the method 1511 can include the video game server, at 1518, identifyinga previous trigger within the historical game content. Further, themethod 1511 can include the video game server, at 1520, determining thecurrent trigger is a same type of trigger as the previous trigger. Inaddition, the method 1511 can include the video game server, at 1522,accessing training game content. Also, the method 1511 can include thevideo game server, at 1524, acquiring machine learning techniques toidentify different triggers from different events from the training gamecontent. Further, the method 1511 can include the video game server, at1526, identifying the current trigger in response to implementing themachine learning techniques on the portion of game content. In someembodiments, the current trigger comprises a trigger type, wherein thetrigger type comprises one of kill, hit, weapon fire, loss of life ofcurrent player, loss of life of another player, and change inammunition.

Referring to FIG. 15C, the associating an event with a trigger can bedone by the video game server in different embodiments. The method 1531can include the video game server, at 1530, determining the group ofevents of the video game. Further, the method 1531 can include the videogame server, at 1532, associating each event in the group of events ofthe video game with each trigger type from the group of trigger types.In addition, the method 1531 can include the video game server, at 1534,receiving second user-generated input. The second user-generated inputprovides an indication of each event associated with each trigger type.

Referring to FIG. 15D, the determining the group of players involved inthe current trigger can be done by the video game server in differentembodiments. The method 1541 can include the video game server, at 1540,implementing image recognition techniques or analyzing audio streams oneach trigger clip of the group of trigger clips. In response, the method1541 can include the video game server, at 1542, identifying the groupof players. Further, the method 1541 can include the video game server,at 1544, analyzing game action during a same time period as each triggerclip of the group of trigger clips. In response, the method 1541 caninclude the video game server, at 1546, identifying the group ofplayers. This analysis can include the video game server determiningfrom game action the location of each player within the video game anddetermining the group of players involved in the current trigger bydetermining the location of each player within the game. Further, thoseplayers within a certain threshold distance from the event/currenttrigger are determined to be involved with the current trigger.

FIGS. 15E and 15F depict illustrative embodiments of methods utilized inthe subject disclosure. Further, the methods depicted in FIGS. 15E and15F can be implemented by a video game server as described herein.Referring to FIG. 15E, a method 1550 is shown illustrating an exampleembodiment for detecting a trigger in one or more audio streams in agaming system. In some embodiments, the detecting of a trigger caninclude detecting a particular sound or utterance or word or statementby a player to identify or otherwise determine that an event within aportion of game content is a trigger. This may be done in any suitablemanner including using artificial intelligence or machine learningtechniques. In an example, a neural network is trained to estimate,within real time, if there is, for example, a laugh or scream from aplayer. After training, a portion of an audio stream is provided to theneural network and the neural network generates a value that correspondsto a probability that the trigger has been detected. A processing systemmay compare the value with a predetermined threshold. If the probabilityexceeds the predetermined threshold, such as 50 percent or 85 percent or95 percent, the characteristic may be identified as a significanttrigger. The training data may include any sort of audio includingconversations, laughter, screaming and other sounds as well, includingsounds that are representative of typical sounds.

A system including an audio processing system such as audio processingsystem 7600 (FIG. 7L) may include a chat audio stream and a microphoneaudio stream. The chat audio stream may include audio corresponding toconversation and other utterances from gamers such as teammates of alocal user. The microphone audio stream may include audio correspondingto conversation and other utterances from the local user, detected bythe microphone adjacent to the local user. In a first use case, a neuralnetwork, or multiple neural networks, may be trained to detect anoccurrence in the chat audio stream or an occurrence in the microphoneaudio stream and, responsive to the detected occurrence, to cause asubsequent action within the gaming system.

In the method 1550, at block 1552, one or more audio streams arereceived from a gaming system. In the example, three audio streamsincluding a game audio stream, a chat audio stream and a player'smicrophone audio stream are received. In other examples, other audiostreams or combined audio streams may be received. In an example,embodiment, the processing of method 1550 is performed at a processingsystem of a player such as a laptop or desktop computer with suitableperipheral devices for audio processing and video processing and one ormore gaming accessories. In other embodiments, the processing of method1550 is performed remotely at a gaming server or on a number of devicesas part of a networked cloud accessible by a player's local device overone or more networks such as the public internet.

At block 1554, one or more audio streams are provided to a neuralnetwork. In the example, the chat audio stream is provided to the neuralnetwork. The neural network has been trained on training data to be ableto recognize and identify a particular data pattern in the audio stream.In the example, the particular data pattern is a laugh or scream utteredby one of the players and reproduced in the data of the chat audiostream. The neural network will produce a value corresponding to aprobability that the particular data pattern has been detected.

At block 1556, the neural network will provide the probability value forfurther evaluation. In exemplary embodiments, the probability value iscompared at block 1558 with a threshold value. Any suitable value, suchas 50 percent or 90 percent, may be used. Moreover, any suitable test oradditional verification may be performed to confirm that the particulartrigger of interest, such as a laugh or a scream by a player, has beendetected. If the probability value does not exceed the threshold,control returns to block 1552 and monitoring of the audio stream oraudio streams continues.

If, block 1558, the probability does exceed the predetermined threshold,at block 1560, a trigger clip including a portion of game content isrecorded. This may be done in any suitable manner. At block 1562, thetrigger clip is distributed, for example, to the current players of thegame. Control returns to block 1552 and monitoring of the audio streamor audio streams continues.

Referring to FIG. 15F, a method 1566 is shown illustrating an exampleembodiment for detecting a trigger in an input stream in a gamingsystem. At block 1568, an input stream is received from the gamingsystem. In exemplary embodiments, the input stream includes an audioinput captured by a microphone, a video input captured by a camera. Insome embodiments, the input stream includes a combination of one or moreaudio streams and one or more video streams. Further, in someembodiments, the input stream includes speech in the audio stream andreceiving the audio stream at block 1568 may include converting speechto text and analyzing the text.

At block 1570, the method 1566 includes receiving a trigger model. In anexample, the trigger model may be provided by a machine learning systemor an artificial intelligence system, or a combination of these. In aparticular embodiment, the trigger model may be generated by a neuralnetwork. In some embodiments, the trigger model may be selected from agroup of predetermined trigger models. The selection of the particulartrigger model may be based on any suitable information including, forexample, identification of the current video game, a genre of the videogame, a location of the gaming device where the audio stream or videostream was capture,

The neural network may be trained on training data including audioinput, training video input or any combination of these. The trainingdata may further include, for example, speech, laughing and screaming bygamers or others. The training data may further include game soundsincluding those generated by the video game itself and sounds that areassociated with a genre of the video game. For example, if the game is awarfare style game, game sounds from other warfare style games may beused as training data. Training data may include video data as well,including for example, facial expressions of game players, hand gesturesand body movements of game players or a combination of these. Thetraining data may include, for example, audio input that is selectedaccording to the video game, the genre of the video game, or a locationof the gaming device, for example. If the gaming device is located inEurope, different training data may be selected compared to a gamingdevice located in South Asia.

At block 1572, the method 1566 detects a current trigger in the inputstream. In some embodiments, this is done substantially in real time. Inthis context, substantially in real time may mean performed with a timedelay that is generally undetectable by the game players, such as asecond or less. If no current trigger is detected at block 1572, controlreturns to block 1568 and monitoring the input stream continues.

At block 1574, a trigger clip is recorded responsive to the detectedcurrent trigger of block 1572. In some embodiments, a predetermined timeperiod before and after the occurrence of the trigger clip is recorded.For example, the current trigger may be determined to have a triggertype such as a particular vocal utterance like a scream or a particularhand gesture. In some embodiments, the duration of the time periodbefore and after the occurrence of the trigger is based on the type oftrigger that is detected at block 1572. In other examples, acharacteristic of the trigger may be identified such as an audio volume,a pitch or vocal inflection, or a word spoken by a player. The timeduration of the time period before and after the occurrence of thetrigger may be based on the trigger characteristic.

In other examples, the current trigger may be associated with aparticular game characteristic of the game at the time of the currenttrigger detected at block 1572. For example, a game characteristic mayinclude a player location of a player in the video game, the occurrenceof some predetermined action by the player or by another player, or anycombination of these. The detecting the current trigger at block 1572may be based on one or more game characteristics. Further, the timeduration of the time period before and after the occurrence of thetrigger may be based on the game characteristic.

In other examples, the current trigger may be associated with an emotionchange of a player involved in the game. For example, the input streammay include a video stream and an audio stream from the player. Based oneither the audio stream alone, the video stream alone or a combinationof the audio stream and the video stream, the method 1566 may includedetecting an emotion change of the player during game play. For example,the video stream may indicate an emotion of happiness based on upraisedarms and a smile of the player. Similarly, the audio stream may indicatean emotion of happiness based on the players audio volume, pitch andrapid rate of speech.

In some embodiments, detecting the current trigger at block 1572 may beperformed in multiple stages. In an example, a first stage is satisfiedbased on identifying a particular type of audio input and a second stageis satisfied based on an identified characteristic of the audio inputsatisfying a threshold. In other examples, a user profile may beaccessed, either locally or from a remote source over a network, and thecurrent trigger may be detected according to the trigger model based onthe user profile. For example, the user profile may include samplesounds for the player, characteristics of sounds of the player, identifya threshold for a sound for the player, or a combination of these. Inthis way, the trigger model may be closely tailored to the player and toeach player of the game.

The illustrations of embodiments described herein are intended toprovide a general understanding of the structure of various embodiments,and they are not intended to serve as a complete description of all theelements and features of apparatus and systems that might make use ofthe structures described herein. Many other embodiments will be apparentto those of skill in the art upon reviewing the above description. Theexemplary embodiments can include combinations of features and/or stepsfrom multiple embodiments. Other embodiments may be utilized and derivedtherefrom, such that structural and logical substitutions and changesmay be made without departing from the scope of this disclosure. Figuresare also merely representational and may not be drawn to scale. Certainproportions thereof may be exaggerated, while others may be minimized.Accordingly, the specification and drawings are to be regarded in anillustrative rather than a restrictive sense.

Although specific embodiments have been illustrated and describedherein, it should be appreciated that any arrangement calculated toachieve the same purpose may be substituted for the specific embodimentsshown. This disclosure is intended to cover any and all adaptations orvariations of various embodiments. Combinations of the aboveembodiments, and other embodiments not specifically described herein,can be used in the subject disclosure. In one or more embodiments,features that are positively recited can also be excluded from theembodiment with or without replacement by another component or step. Thesteps or functions described with respect to the exemplary processes ormethods can be performed in any order. The steps or functions describedwith respect to the exemplary processes or methods can be performedalone or in combination with other steps or functions (from otherembodiments or from other steps that have not been described).

Less than all of the steps or functions described with respect to theexemplary processes or methods can also be performed in one or more ofthe exemplary embodiments. Further, the use of numerical terms todescribe a device, component, step or function, such as first, second,third, and so forth, is not intended to describe an order or functionunless expressly stated so. The use of the terms first, second, thirdand so forth, is generally to distinguish between devices, components,steps or functions unless expressly stated otherwise. Additionally, oneor more devices or components described with respect to the exemplaryembodiments can facilitate one or more functions, where the facilitating(e.g., facilitating access or facilitating establishing a connection)can include less than every step needed to perform the function or caninclude all of the steps needed to perform the function.

In one or more embodiments, a processor (which can include a controlleror circuit) has been described that performs various functions. Itshould be understood that the processor can be multiple processors,which can include distributed processors or parallel processors in asingle machine or multiple machines. The processor can be used insupporting a virtual processing environment. The virtual processingenvironment may support one or more virtual machines representingcomputers, servers, or other computing devices. In such virtualmachines, components such as microprocessors and storage devices may bevirtualized or logically represented. The processor can include a statemachine, application specific integrated circuit, and/or programmablegate array including a Field PGA. In one or more embodiments, when aprocessor executes instructions to perform “operations”, this caninclude the processor performing the operations directly and/orfacilitating, directing, or cooperating with another device or componentto perform the operations.

The Abstract of the Disclosure is provided with the understanding thatit will not be used to interpret or limit the scope or meaning of theclaims. In addition, in the foregoing Detailed Description, it can beseen that various features are grouped together in a single embodimentfor the purpose of streamlining the disclosure. This method ofdisclosure is not to be interpreted as reflecting an intention that theclaimed embodiments require more features than are expressly recited ineach claim. Rather, as the following claims reflect, inventive subjectmatter lies in less than all features of a single disclosed embodiment.Thus the following claims are hereby incorporated into the DetailedDescription, with each claim standing on its own as a separately claimedsubject matter.

What is claimed is:
 1. A method, comprising: receiving, by a processingsystem including a processor, a gaming system output stream during avideo game, the gaming system output stream including audio outputcaptured by a microphone, video output captured by a camera, or acombination thereof; detecting, by the processing system, a currenttrigger in the gaming system output stream during the video game;recording, by the processing system, a trigger clip of game content,wherein the trigger clip comprises a portion of game content thatincludes the current trigger; and providing, by the processing system,the trigger clip to a group of gaming devices over a communicationnetwork, wherein each gaming device of the group of gaming devices isassociated with one player of the video game.
 2. The method of claim 1,wherein the detecting a current trigger comprises: providing, by theprocessing system, the gaming system output stream to a neural network;and receiving, by the processing system, an indication that the currenttrigger has been detected.
 3. The method of claim 2, wherein thereceiving an indication that the current trigger has been detectedcomprises: receiving, by the processing system, from the neural networka value that corresponds to a probability that the current trigger hasbeen detected.
 4. The method of claim 3, further comprising: comparing,by the processing system, the value that corresponds to a probabilitywith a predetermined threshold; and determining, by the processingsystem, that the current trigger has been detected responsive to thecomparing.
 5. The method of claim 2, wherein the neural network has beentrained on training data from one or more video games.
 6. The method ofclaim 5, wherein the training data comprises data corresponding to vocalutterances and non-vocal utterances of players of the one or more videogames.
 7. The method of claim 6, wherein the training data comprisesdata corresponding to conversations, laughter and screaming of playersof the one or more video games.
 8. The method of claim 1, wherein thedetecting a current trigger in the gaming system output streamcomprises: identifying, by the processing system, one of a laugh or ascream by a player of the video game in the gaming system output stream.9. The method of claim 1, wherein the receiving a gaming system outputstream comprises: receiving, by the processing system; a plurality ofaudio streams including a chat audio stream, a game audio stream and amicrophone audio stream; and detecting, by the processing system, thecurrent trigger in one of the chat audio stream and the microphone audiostream.
 10. The method of claim 1, further comprising: determining, bythe processing system, a group of players involved in the currenttrigger; recording, by the processing system, the trigger clip of gamecontent for each player in the group of players, resulting in a group oftrigger clips, wherein each trigger clip of the group of trigger clipscomprises a portion of game content that includes the current triggerfrom a perspective of each player of the group of players; andproviding, by the processing system, the group of trigger clips to thegroup of gaming devices over the communication network, wherein eachgaming device of the group of gaming devices is associated with oneplayer of the group of players.
 11. A method comprising: receiving, by aprocessing system including a processor, an input stream during a videogame, the input stream including audio input captured by a microphone,video input captured by a camera, or a combination thereof; applying, bythe processing system, a trigger model to the input stream, wherein thetrigger model is generated by a neural network trained on a trainingaudio input, a training video input, or a combination thereof;detecting, by the processing system according to the trigger model, acurrent trigger in the input stream during the video game, wherein thecurrent trigger comprises an audio trigger, a video trigger orcombination thereof; and recording, by the processing system, a triggerclip of game content, wherein the trigger clip comprises a portion ofgame content for a time period before and after the current trigger. 12.The method of claim 11, wherein the training audio input includesspeech, laughing, and screaming.
 13. The method of claim 11, wherein thetraining video input includes facial expressions, hand gestures, bodymovements, or a combination thereof.
 14. The method of claim 11, whereinthe time period before and after the current trigger is selected from agroup of time periods according to a type of current trigger.
 15. Themethod of claim 11, comprising: identifying a characteristic of thecurrent trigger resulting in a trigger characteristic, wherein the timeperiod before and after the current trigger is selected from a group oftime periods according to the trigger characteristic.
 16. The method ofclaim 15, wherein the trigger characteristic includes a volume, a pitch,or a word.
 17. The method of claim 11, wherein the time period beforeand after the current trigger is selected from a group of time periodsaccording to a game characteristic.
 18. The method of claim 11, whereinthe detecting the current trigger in the input stream during the videogame is based on detecting an emotion change of a player from videoinput of the player and audio input captured from the player.
 19. Themethod of claim 11, comprising: selecting, by the processing system, thetrigger model from a group of trigger models according to the videogame, a genre of the video game, a location of a gaming device where theaudio input or video input was captured, or a combination thereof. 20.The method of claim 11, further comprising capturing speech by themicrophone as the audio input; and converting the speech to text andanalyzing the text, wherein the detecting the current trigger in theinput stream during the video game is based on the analyzing the text.